US20040223003A1 - Parallel pipelined merge engines - Google Patents

Parallel pipelined merge engines Download PDF

Info

Publication number
US20040223003A1
US20040223003A1 US10/864,609 US86460904A US2004223003A1 US 20040223003 A1 US20040223003 A1 US 20040223003A1 US 86460904 A US86460904 A US 86460904A US 2004223003 A1 US2004223003 A1 US 2004223003A1
Authority
US
United States
Prior art keywords
merge
engine
processor
pixel
bus
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10/864,609
Inventor
Alan Heirich
Laurent Moll
Mark Shand
Albert Tam
Robert Horst
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tandem Computers Inc
Original Assignee
Tandem Computers Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tandem Computers Inc filed Critical Tandem Computers Inc
Priority to US10/864,609 priority Critical patent/US20040223003A1/en
Publication of US20040223003A1 publication Critical patent/US20040223003A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/005General purpose rendering architectures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00Indexing scheme for image generation or computer graphics
    • G06T2210/52Parallel processing

Definitions

  • the present invention relates to image generation in general and in particular to rendering engines for generating images that might be needed to be generated in real-time response to an input of objects in a scene.
  • Rendering is the process of computing a two-dimensional viewpoint dependent image of a three-dimensional geometric model. Rendering is computationally intensive, since a typical geometric model might contain objects that collectively comprise millions of polygons.
  • a geometry stage first converts mathematical descriptions of arbitrarily complex objects into a collection of polygons, typically triangles, so that subsequent rendering stages only need to deal with simply polygons. The rendering stages then need to simulate the transport of light at various wavelengths emitted from numerous light sources in the geometric model and scattered by surfaces of objects in the geometric model.
  • ray tracing Two common methods of handing light-object interactions are known as “ray tracing” and “radiosity” methods. If those methods are implemented in software, the software can take from minutes to days to produce a single high quality “photorealistic” image of a large geometric model.
  • a typical photorealistic image might require a resolution of 2000 pixels by 1000 pixels and contain a million or more separately specified scene elements (i.e., objects or light sources). While days might be available for rendering some images, many applications require real-time rendering of video frames, such as computer games, flight simulators, scientific visualizers, computer-aided design/computer-aided engineering (CAD/CAE) applications and medical data visualizers.
  • CAD/CAE computer-aided design/computer-aided engineering
  • the delay allowed between the receipt of a geometric model and the output of the corresponding image is determined, in part, by the frame rate.
  • the frame rate can be as low as one frame per minute, so a viable rendering system for that application must produce at least one image per minute if it is to run at an acceptable rate.
  • the rendering system is expected to run at thirty frames or more per second. In some thirty frame/second applications, the delay might need to be even less, as might be the case for a flight simulator or computer game having feedback.
  • an operator of the system provides inputs, such as commands to move up, down or sideways, and the corresponding changes to the geometric model are provided to a rendering engine.
  • the rendering engine might have to generate an image from a geometry model in even less time than a frame period.
  • a graphics pipeline comprises a few well-understood stages: the geometry stage, the rendering stage and the composition stage. In a rendering engine with a graphics pipeline, some or all of these stages are implemented in parallel.
  • a database of geometric objects (usually triangles) is read from host memory and transformed from a world coordinate system (“object space”) into a view dependent coordinate system (“screen space”).
  • object space world coordinate system
  • screen space view dependent coordinate system
  • the geometry stage might also perform the conversion of complex objects into simplified objects comprising polygons. Because all polygons can be represented as triangles, and processing triangles is simpler than processing polygons with more than three sides, nearly all existing rendering engines limit geometric objects to triangles.
  • each vertex is defined by a coordinate (x, y, z) and a surface normal vector (dx, dy, dz) and might also be defined by surface material properties (such as a coefficient of reflectivity and transmissivity) and the surface may be bound to textures.
  • the geometry stage converts the vertex coordinates and normals from object space representations into their screen space counterparts (with a resolution being a function of the resolution of the output image) and discards objects that fall entirely outside the perimeter of the screen (the “view surface”).
  • the geometry stage also performs lighting calculations, as needed, at the vertices for use by later stages that will interpolate these results at individual pixels.
  • the rendering stage calculates the pixel color values that correspond to those polygons.
  • the composition stage combines the results from the parallel rendering engines to form the final image.
  • load balancing is to ensure that the work to be done is evenly distributed over all of the parallel threads, to avoid having some threads idle while others are still processing. Thread independence is desirable, since independent threads will not be held up waiting for other threads to reach a certain point in their processing. Some thread dependence might be unavoidable, but such dependence might not always result in additional delays.
  • sort-middle One approach to dividing up the work among a plurality of threads is known as the “sort-middle” architecture.
  • the screen space is divided into tiles and each tile is assigned a bin, where the bin corresponds to the work one rendering thread will perform.
  • the objects comprising the geometric stage's output are then allocated to the bins, with each object only being allocated to the bins of the tiles that the object intersects.
  • a Z-comparison depth comparison
  • Shading and blending calculations are then performed to compute the amount of light reflected at each wavelength (typically the three wavelengths r, g and b) by the closest object in each pixel, after textures have been taken into account.
  • the result of this calculation is the color value (e.g., r, g and b values) of the pixel.
  • sort-last architecture wherein, instead of subdividing the screen, the objects are allocated randomly to the bins and each thread does a Z-comparison on the objects in its bin. Each thread computes an entire screen image, albeit an incorrect one containing less than all of the objects in the geometric model.
  • the Z (depth) information is stored along with the pixel color values.
  • the composition stage in a sort-middle architecture simply combines the subimages for each tile into the final image.
  • the composition stage in a “sort-last” architecture uses the Z information to merge all of the part (“incorrect”) images to produce a single correct image.
  • a merge engine is used to combine the part images by identifying which objects in which part images overlap which objects in other part images. In some cases, where there are many parallel threads, the composition stage might be done in parallel.
  • Sort-last architectures typically provide better load balancing than sort-middle architectures, since some sort-middle threads might be allocated tiles that contain few or no objects.
  • Inexpensive rendering cards for personal computers are able to render about one million triangles per second. However, if a thirty frame/second video image is needed, the most complicated model that the rendering card could process in real-time would be limited to no more than about 33,000 triangles. Some computer games might be playable with such a limited number of triangles, but generating a photorealistic image with 33,000 triangles is difficult or impossible.
  • Many CAD/CAE applications such as those used by auto makers require finite element models with a hundred thousand to a million elements. To render an automotive model with a hundred thousand to a million triangular elements in real-time at thirty frames per second, a rendering engine would need to process three to thirty million triangles per second. A model with five million elements, which may be realistic within a few years, would require a rendering engine that can process 150 million triangles per second.
  • the bandwidth required at the input to the geometry stage of the graphics pipeline can be estimated from the model size. If a rendering card can process a large number of triangles/second, it cannot be used to its full capacity unless the communication channel to that rendering card can carry all of the data specifying those large numbers of triangles.
  • Each triangle is represented by at least three vertices, and each vertex by six floating point numbers (a coordinate and a surface normal vector) for a total of seventy-two bytes in the typical specification of a triangle.
  • Surface normal vectors are usually defined at each vertex when the triangle represents a facet of a tessellated surface such as a spherical object.
  • Another data reduction method is to use “triangle strips” which eliminate redundant vertex definitions, and require one additional vertex for each additional triangle. Worst case, a rendering card that can process one million triangles per second would require a 72 megabyte/second (MBS) communication bandwidth.
  • PCI bus Peripheral Component Interconnect Express
  • 72 MBS is about the achievable capacity of the current PCI bus.
  • a system that renders three million triangles per second requires a bandwidth of roughly 216 MBS. This bandwidth should be achievable over newer buses, such as Intel's Accelerated Graphics Port (AGP) bus.
  • AGP Accelerated Graphics Port
  • the present invention overcomes several disadvantages of the prior art methods and apparatus for generating images.
  • the image generator is organized into a plurality of rendering engines, each of which renders an image of a part scene and provides the part image to a merge engine associated with that rendering engine.
  • the image is a part image in that it usually contains less than all of the objects in the image to be rendered.
  • the merge engine merges the part image from its associated rendering engine with the part image provided by a prior merge engine (the prior neighbor in a merge engine sequence) and provides the merged part image to a next merge engine (the next neighbor in a merge engine sequence).
  • One or more merge engines are designated as output merge engines and these output merge engines output a merged part image that is (a portion of) the ultimate output of the image generator, the full rendered image.
  • Each merge engine performs its merge process on the pixels it has from its rendering engine and from its prior neighbor merge engine, without necessarily waiting for all of the pixels of the part image or the merged part image.
  • One advantage of the image generator is that the operations of the merge engines are pipelined, since they operate on pixels as they arrive, if an operation can be done on the arrived pixels, instead of waiting for all pixels to arrive.
  • the merge sequence is dynamic.
  • the merge engines are coupled to a switch network where the switch network determines which merge engines are prior neighbors and next neighbors to which merge engines.
  • the switch network makes those determinations based on the objects being provided to the individual rendering engines.
  • the image generator can be used for a variety of image compositing and “two-and-a-half-D” applications, which combine 3D rendering, 2D overlays and special operations including color correction, dissolves, blending and other transformations.
  • FIG. 1 is a block diagram of an image generator according to one embodiment of the present invention.
  • FIG. 2 is a block diagram of a merge engine as might be used in the image generator of FIG. 1.
  • FIG. 3 is a block diagram illustrating an image composition process according to the present invention.
  • FIG. 4 is a block diagram of a parallel image generator implemented with multiple workstations.
  • FIG. 5 is a block diagram illustrating the use of a switch network.
  • FIG. 6 is a block diagram of a specific implementation of a merge engine node in a parallel pipeline graphics processor.
  • FIG. 7 is a more detailed block diagram of the merge engine shown in FIG. 6.
  • FIG. 8 is a timing diagram illustrating the pipelining of the graphics process performed by the processor shown in FIGS. 6-7.
  • FIG. 1 The construction and operation of a basic image generator according to one embodiment of the present invention is shown in FIG. 1.
  • a graphics system 10 is shown with an image generator 12 as might be used to render an image from an object description file 14 and present a rendered set of pixel color values to a display 16 or an image store 18 .
  • Object description file 14 can be any type of data file or stream of expressions that indicate where objects and light sources are in a scene, as well as other parameters needed to render an image according to conventional methods. While only two output devices are shown, the output of image generator 12 might be used in other ways, such as being fed into another computer system or program for further analysis and/or processing of the rendered image, or any other use typically made of rendered images.
  • the image generator itself is shown comprising an image controller 20 , a plurality of rendering engines 22 , a plurality of merge engines 24 and a frame store 26 .
  • a paired set of a rendering engine and an associated merge engine might be implemented on one workstation, as described below in reference to later figures, but this description of the basic system is not limited to any particular limitation on the allocation of the physical hardware.
  • an index is indicated, as in “ 22 ( i )” or “ 22 i ” to identify the i-th rendering engine.
  • rendering engines 22 ( 1 ), 22 ( 2 ), 22 (N ⁇ 1) and 22 (N) are shown and, for N>4, rendering engines 22 ( 3 ) through 22 (N ⁇ 2) are implied.
  • image controller 20 provides for an input of an object description file to image controller 20 and for transfer of subsets of the objects to rendering engines 22 , e.g., object(s) OBJ 1 to image controller 22 ( 1 ), object(s) OBJ 2 to image controller 22 ( 2 ), etc., in a “sort-last” or “sort-middle” manner.
  • image controller 20 might be configured to send only one object to each rendering engine 22 , i.e., the description OBJ i is the description of just one object.
  • image controller 20 might send multiple objects to one or more rendering engine, to load balance (i.e., give each rendering engine roughly the same amount of work to do) or to distribute all of the objects at once when there are more than N objects in the scene.
  • Image controller 20 is shown in FIG. 1 as one unit, but in some embodiments, image controller 20 is implemented as a distributed application. In those embodiments, the application might be distributed over a collection of machines that are the same machines that are hosting the rendering engines 22 and merge engines 24 . If that is the case, then host bus 32 might be replaced with a set of host busses, one per host, to connect the host component of the image controller to its corresponding rendering engine and merge engine.
  • a data path is provided between each rendering engine 22 and its corresponding merge engine 24 for communicating a part image from the rendering engine to the merge engine, wherein the part image PI i communicated from rendering engine 22 ( i ) to merge engine 24 ( i ) is a rendered image of the object(s) described by OBJ i .
  • Each of the merge engines 24 is also provided with a data path to its nearest prior neighbor and its nearest next neighbor. Thus, for merge engine 24 ( i ), a data path is provided from merge engine 24 ( i ⁇ 1) and a data path is provided to merge engine 24 ( i +1).
  • Merge engines are logically arranged in a ring, so that the nearest prior neighbor to merge engine 24 ( 1 ) is merge engine 24 (N) and the nearest next neighbor to merge engine 24 (N) is 24 ( 1 ).
  • N elements numbered from 1 to N
  • “i ⁇ 1” and “i+1” refer to the i-th elements' nearest neighbor
  • i ⁇ 1 refers to the N-th element
  • i+1 refers to the first element.
  • the data paths between merge engine neighbors can be direct connections, but the embodiment shown in FIG. 1 uses a high-speed pixel bus 28 to carry the traffic between merge engines.
  • One suitable high-speed bus is the ServerNet® network developed by Tandem Computers Incorporated.
  • the ServerNet® network is described in Heirich, Garcia, Knowles and Horst, “ServerNet-II: A Commodity Interconnect for Scalable High Performance Cluster Computing”, which is incorporated by reference herein for all purposes.
  • One of the merge engines 24 is designated as the output merge engine.
  • the designated merge engine is merge engine 24 (N), which is provided with a data path to a frame buffer 26 and holds the output of image generator 12 , which is a merged image of all of the part images PI.
  • the frame buffer of rendering engine 24 (N) is used instead of a dedicated frame buffer such as frame buffer 26 .
  • Some implementations might also use more than one output merge engine. Other connections might be provided but not shown as described herein.
  • one advantage of the arrangement shown in FIG. 1 is that the merger of part images is done in a pipeline, with pixel values forming the part images and merge images streaming from the rendering engines and merge engines as they become available.
  • the merge engine completes the merger of a block of pixels streamed from its corresponding rendering engine with a block of pixels streamed from the merge engine's nearest prior neighbor, it passes the merged block on to its nearest next neighbor.
  • the block of pixels can range in size from one pixel to the number of pixels in the image, but as the block size approaches the number of pixels in the image, the benefits of pipelining diminish, and are not obtained when the block size is the size of the image. In one prototype, the image size is 307,200 pixels (640 by 480 pixels) and the block size is 8 pixels.
  • the link between 24(N) and 24 ( 1 ) is omitted in some configurations that do not require recirculation of intermediate image results, depending on the nature of the rendering process being employed. However, in general, the link is included.
  • each merge engine 24 acquires image data from its corresponding rendering engine 22 through a render-merge interface 30 .
  • rendering engines 22 are standard graphics accelerator cards with PCI interfaces
  • render-merge interface 30 can be a PCI interface, but a private digital interface that performs better than PCI interfaces might be preferred.
  • Each merge engine 24 receives pixel color values, and other data, from its corresponding rendering engine 22 and also receives image data from its nearest prior neighbor via high-speed pixel bus 28 .
  • pixel bus 28 is implemented using a ServerNet® network
  • the interface to pixel bus 28 could be implemented using a ServerNet® “Colorado” application-specific integrated circuit (ASIC) developed by Tandem Computers Incorporated for handling communications over a ServerNet® network.
  • ASIC application-specific integrated circuit
  • a merge engine 24 ( i ) combines its two streams of image data to produce a third stream, which is sent through an outgoing connection on ASIC to the next merge engine 24 ( i +1) in the pathway. At one or more points in the pathway, a merge engine will push the combined imagery onto a frame buffer to allow for display of the image.
  • the ASIC is documented in detail in the “Colorado External Specification”, Tandem Computers Document Number 122219-X05 dated Oct. 14, 1997, which is incorporated by reference herein for all purposes.
  • Image generator 12 might advantageously be built from readily available graphics components, to reduce the design efforts and costs of an image generator.
  • rendering engines 22 could be standard graphics accelerator cards and merge engines 24 could incorporate an existing driver for a ServerNet® network interface, such as the Colorado ASIC.
  • a typical, off-the-shelf commodity graphics card has a PCI bus interface (a standardized bus used for inter-board communications in desktop computers) for receiving object descriptions and can render a few million triangles per second (objects are typically broken down into collections of triangles for rendering).
  • An image generator is usually “parallelized” to overcome limits in of processing speed in each thread or limits in bandwidth into, out of, or internal to, the thread.
  • processing speed is the stricter constraint
  • the number of “rendering threads” (“N” in the examples described herein) is determined by dividing the maximum number of triangles/second needed in the application by the number of triangles/second processable by one thread (which is often limited to the number of triangles/second processed by the rendering card that forms part of the thread).
  • the output bandwidth of the rendering stages is a constraint.
  • each rendering engine will produce a stream of thirty frames/second of video.
  • Each pixel on the output might comprise 24 bits of color information (8 bits each for red, green and blue), 16 to 32 bits of Z (depth) information, and possibly an additional eight bits of alpha (transparency) data, totaling from 40 to 60 bits of data.
  • 12 bits/color might be the norm, resulting in a minimum of 52 bits.
  • the typical format is 30 frames/second with each frame comprising 640 pixels by 480 pixels, for a total of 307,200 pixels per frame.
  • NTSC rendering engines would output between 44 and 70 MBS of data.
  • television will expand to the HDTV format, with its approximately 2000 ⁇ 1000 pixel frames, requiring a bandwidth of between 286 and 455 megabytes/second.
  • FIG. 2 is a block diagram showing one merge engine 24 ( i ) in further detail.
  • Merge engine 24 ( i ) is logically arranged into merge engine logic 100 and a merge engine interface circuit 102 for connecting neighboring merge engines and merge engine interface circuit 102 uses an ASIC for handling the communication between merge engines over a network such as a ServerNet® network.
  • ASIC application-specific integrated circuit
  • Other connection approaches could be used, but using an ASIC that is used for other applications has well-known advantages, such as the fact that it eliminates the need to design communications logic specifically for the merge engine communication and having a broader base of uses for any given ASIC distributes the design costs over more chips, thus reducing the per-chip design costs.
  • the N pipeline stages of an image generator can be implemented very economically.
  • FIG. 2 Three interfaces are shown in FIG. 2. At the top, an ASIC 102 connects the merge engine to its nearest neighbors over pixel bus 28 . The other two interfaces are a part image bus 30 between the merge engine and its corresponding rendering engine and a host PCI bus 32 between the merge engine and a host which is used for device initialization and host control.
  • connection to pixel bus 28 are labeled “x” and “y”, as those are the two channels provided by the Colorado ASIC.
  • the Colorado ASIC makes the design of the merge engine shown herein quite cost effective, especially since the ASICs can be used essentially unchanged.
  • Each ASIC has two ports (termed “x” and “y”) that are usually used redundantly to provide fault tolerance.
  • the “x” and “y” points are used for data flow in one direction, but in other embodiments, those parts, which are bi-directional, are used to carry data in both directions, giving higher effective bandwidth and frame rate.
  • Merge engine logic 100 comprises various elements for communication between a function-unit core (core 110 ) 110 and the three interfaces.
  • Core 110 operates on pixel streams arriving at the merge engine over pixel bus 28 (from other merge engines) and over part image bus 30 from an associated rendering engine. In particular, core 110 performs operations on pairs of pixels.
  • Core 110 is connected to ASIC 102 through a PCI interface unit 120 .
  • Incoming pixel data flows from PCI interface unit 120 through an arbiter 122 , an input buffer 124 and a decoder 126 , while outgoing pixel data flows from core 110 to PCI interface unit 120 through an encoder 130 , an output buffer 132 and a sender 134 .
  • image data When image data is transferred from one merge engine to its nearest next neighbor, the image data is transferred as a message of a fixed multiple of the ServerNet® network packet data size.
  • ASIC 102 When a message arrives on the input to ASIC 102 , ASIC 102 passes the message into input buffer 124 via PCI interface unit 120 and arbiter 122 .
  • the PCI bus 136 between ASIC 102 and merge engine logic 100 is not required, as other bus protocols might be used, but with a standard PCI interface to merge engine logic 100 , the existing interface of the Colorado ASIC need not be redesigned.
  • ASIC 102 At the same time that the message arrives from ASIC 102 , another message of the same length arrives over the part image bus 30 into a host buffer 140 .
  • Core logic handles flow control, packet synchronization, and data transfers between the merge engine and rendering engine. With the arrival of these two messages, core 110 can perform pixel-by-pixel computations on these two messages to produce a result message with the same fixed length. The details of the possible computations are described below with reference to core 110 operation.
  • the result message is sent to ASIC 102 for transmission to the merge engine 24 ( i +1).
  • Merge engine 24 ( i ) usually operates on two input streams of pixel data: one from merge engine 24 ( i ⁇ 1) and one from rendering engine 22 ( i ) and usually produces a single output stream of pixel data, which is sent downstream to the merge engine 24 ( i +1).
  • the merge engine may use only a single input stream, may obtain additional data from a host, and/or may write its output stream to the host.
  • These operating modes are defined by control registers in the merge engine that may be set independently by the host or by control tokens arriving over pixel bus 28 .
  • Input pixel data is written into input buffer 124 , which is a circular buffer, unless access to input buffer 124 is denied by the arbiter 122 .
  • Decoder 126 removes data from input buffer 124 and decompresses the data, one pixel at a time, at the request of core 110 . In some implementations, compression is not used and decoder 126 and encoder 130 are not needed. Decoder 126 informs arbiter 122 whether input buffer 124 has room to accept new incoming data and arbiter 122 uses this information to decide when to deny access to input buffer 124 .
  • Host buffer 140 unit supplies data to core 110 , one pixel at a time, at the request of the core 110 .
  • Host buffer 140 contains data from up to four separate segments of host address space, in blocks that are sufficiently large to ensure that data is always available when requested by core 110 .
  • Host buffer 140 is bi-directional, in that it will also buffer data being sent from core 110 .
  • core 110 requests data for one pixel from decoder 126 and one pixel from host buffer 140 . It uses that data to compute values for one new pixel, which core 110 then provides to encoder 130 .
  • Encoder 130 when requested by the core 110 , compresses the data for a pixel and inserts the result into output buffer 132 , which is also a circular buffer.
  • Encoder 130 informs sender 134 when output buffer 132 contains enough data to send downstream. When this happens, sender 134 generates the appropriate PCI transactions to move the data from output buffer 132 to the input buffer 124 of merge engine 24 ( i +1) via ASIC 102 .
  • merge engine 24 ( 1 ) usually reads an input stream of pixel data and passes it downstream to merge engine 24 ( 2 ) without accepting any pixel data from the upstream direction.
  • a merge engine will read an input stream from its nearest prior neighbor without taking any data from its rendering engine.
  • the final merge engine 24 (N) will write an output stream of pixel data to memory on a frame buffer of rendering engine 24 (N) or frame buffer 26 , but will not send data downstream.
  • every merge engine except 24 (N) will write an output stream of Z information to the host rendering engine and also to its next nearest neighbor.
  • control messages might be sent along the merge pathway to modify status, control and address registers within the merge engines. These registers are shown in FIG. 2 as registers 144 and are described in further detail below.
  • Each merge engine is assigned a group identifier, GID, and every control message contains a GID.
  • GID group identifier
  • Arbiter 122 implements flow control for input buffer 124 , telling decoder 126 when fresh data is available, and preventing the upstream sender 134 from overwriting active data that has not yet been decoded. Arbiter 122 is also responsible for recognizing control messages when they arrive and routing them to the appropriate logic.
  • Input buffer 124 is a circular buffer that holds pixel data that is waiting to be decoded by decoder 126 (if compression is used) and consumed by the core 110 .
  • Core 110 will most likely consume data more quickly than pixel bus 28 can deliver it, so there is little chance of overflow in normal operation and input buffer 124 can be a simple double buffer. It will accept a data and address interface from arbiter 122 and export an interface to decoder 126 that will provide one byte at a time in FIFO order.
  • Decoder 126 unit decompresses pixel data (if compression is used).
  • One suitable compression scheme is that described in U.S. Pat. No. ______ (U.S. patent application No. ______ filed on the same day as the instant application with attorney docket number 10577-459 or 010577-045900US). If compression is used, decoder 126 decompresses data from input buffer 124 , one byte at a time, until it accumulates the data for an entire pixel (usually between 5 and 8 decompressed bytes). Decoder 126 then passes the decompressed result to core 110 when core 110 requests a pixel. Flow control between decoder 126 and core 110 is implemented by a handshake.
  • Core 110 is the computational heart of merge engine 24 .
  • core 110 takes as input the data for a single pixel from decoder 126 and data for a single pixel from host buffer 140 . Under some circumstances, core 110 may take additional data from host buffer 140 .
  • Core 110 produces result data for a single pixel, which may be sent to encoder 130 , host buffer 140 or both. Under some circumstances, the result data might be combined with additional data from host buffer 140 in a two-stage computation.
  • Flow control is implemented by handshakes between decoder 126 and core 110 , host buffer 140 and core 110 , core 110 and encoder 130 , and core 110 and host buffer 140 .
  • Core 110 should not compute the next pixel until it receives handshakes from decoder 126 and host buffer 140 (if two input streams are active) to confirm the data is ready; after core 110 sends the data for the new pixel to encoder 130 , core 110 should not generate more pixel data until it receives acknowledgments from encoder 130 (and host buffer 140 , if it is an active destination).
  • a Z comparison operation in which one of the two input pixels is discarded based on their relative Z values
  • a blend operation in which transparent pixels are blended using a variety of blending formulas.
  • Other pixel operations can easily be added to support commercially important markets, including image compositing and digital special effects.
  • Core 110 processes data that corresponds to image pixels and Z (depth) information.
  • a pixel is usually defined as 24 bits, 8 bits each for R, G, and B color data, or as 32 bits, with an additional 8 bits of “alpha” transparency data.
  • the Z information for a pixel is usually defined by a 16, 24, or 32-bit integer. In a specific implementation, the design of core 110 should not preclude supporting higher numbers of bits in either pixel or Z data.
  • the computations of merge engine logic 100 are defined in terms of individual pixels, it should be understood that the computation would proceed most efficiently when blocks of pixels are processed together.
  • Core 110 will perform different computations in different operating modes. In some circumstances, core 110 may not even perform computation on the data, but may pass the data through unchanged.
  • One operation is a Z comparison, which is expected to comprise nearly 95% of the operations performed by a core. for this operation, given two pixels P1 (R1, G1, B1, Z1) and P2 (R2, G2, B2, Z2), if (Z1 ⁇ Z2) then the result pixel is P1, otherwise the result pixel is P2.
  • a more complex operation is “blending” to handle atmospheric effects and transparency.
  • Alpha blending is used to render transparent surfaces and is the basis for volume rendering. Instead of computing a pixel color from a single polygon, a rendering system can treat polygons as semitransparent objects (for example, the windshield of an automobile) and compute pixel colors by blending the colors of a number of polygons along a single line of sight. Each polygon has an associated “alpha” coefficient, between zero and one, that determines its relative transparency.
  • the blending operation is not commutative, which means that it must be computed in order along a line of sight. This poses some challenges for any parallel rendering architecture because it means that the relative positions of objects must be maintained through the pipeline.
  • the result pixel, PR may be computed as:
  • PR ( F ( A 1 ,R 1, R 2), F ( A 1 ,G 1, G 2), f ( A 1 ,A 1 ,A 1 ,A 2), MIN ( Z 1 Z 2)),
  • a data pass-through operation might be provided, wherein a merge engine passes a stream of data from either input (host PCI or ASIC) to its output, unmodified.
  • Merge engine 24 ( 1 ) will often have to do this in order to feed the head of a merge pipeline.
  • pixel data or Z data can be passed independently and from either input.
  • Another operation is data broadcast, as might be used in connection with a multi-pass algorithm to render transparent surfaces, it will be necessary to broadcast data from one frame buffer to some or all other frame buffers.
  • a model containing transparent components can be rendered by first rendering all of the opaque components, merging the results, and then reloading all of the Z buffers with the merged Z result before proceeding to render the transparent components.
  • the pixel data and/or the Z buffer data can be written to the host from data arriving over the ASIC input.
  • Another operation typically performed by merge engine 24 (N), is the write out of results to the final frame buffer. This operation might be useful in other stages of the pipeline, or for other operations that would require circulating data multiple times between a merge engine and its host.
  • the merge engine is created using FPGA's as described above, the operations performed by the merge engine can be updated as needed by reprogramming the FPGA's.
  • One implementation of a merge engine uses a custom-designed board that plugs into a PCI slot of a workstation, where the merge engine board contains a ServerNet® Colorado ASIC and commodity and custom circuitry to perform pixel compositing, blending and other operations, as well as to transfer data in and out of workstation memory.
  • the merge engines are connected in series into a (logical or physical) ring topology. In a simple, cost-effective configuration, these boards are connected in a daisy-chained ring among a set of workstations. In more complex configuration, the boards are connected to a switch fabric to increase throughput, provide fault tolerance or to support specialized applications like volume visualization.
  • encoder 130 that element accepts one pixel of data from core 110 , compresses it, and writes the compressed result to output buffer 132 .
  • Encoder 130 exports a flag to sender 134 to request transmission of the contents of output buffer 132 downstream to the input buffer of the next merge engine.
  • Encoder 130 also accepts a flag from sender 134 that indicates which half of output buffer 132 encoder 130 should write into. If encoder 130 recognizes that it is about to overwrite data that has not yet been sent downstream (which should only occur if sender 134 is blocked from sending downstream) then encoder 130 will not execute the handshake with core 110 and thus will effectively block further computation in core 110 . Otherwise, encoder 130 sends an acknowledgment to core 110 when encoder 130 has encoded the pixel data and written it into output buffer 132 .
  • Output buffer 132 is a double buffer similar to input buffer 124 .
  • the contents of output buffer 132 are accessible to sender 134 so that sender 134 can generate the PCI transactions required to transfer the contents to the input buffer of the next merge engine.
  • Sender 134 is a data movement mechanism, such as a DMA engine, that pushes data, one packet at a time, from output buffer 132 to PCI bus 136 between sender 134 and ASIC 102 . Since sender 134 uses normal PCI arbitration to gain access to PCI bus 136 , the flow control mechanisms described previously for arbiter 122 will ensure that sender 134 does not overwrite active downstream data. Flow control in the reverse direction is accomplished by exporting a bit flag to tell encoder 130 which half of output buffer 132 to write into. In the switched configurations described below, sender 134 is responsible for splitting the outgoing data stream across two links.
  • a set of control, status and address registers are provided in each merge engine 24 .
  • the registers are always be accessible to the host through host PCI bus 32 , although the design of merge engine 24 might include a constraint that a host is not allowed, or expected, to modify some of the registers while computations are occurring in core 110 .
  • the registers are also accessible to other merge engines, through the use of control messages passed from merge engine to merge engine over pixel bus 28 . If pixel bus 28 is a ServerNet® network bus, this feature can be used to implement a variety of control operations at high speed without involving the individual hosts.
  • the registers are logically arranged in a PCI address space, so they can be read by an adjacent merge engine via ASIC 102 using standard PCI commands.
  • Host buffer 140 transfers pixel data from up to four independent segments of host PCI address space, buffers it, and assembles it into pixel format for use by core 110 .
  • Host buffer 140 , or host PCI interface 142 maintains and increments appropriate address registers so that the transfers can occur in bursts in a predetermined order.
  • the data is transferred to core 110 when core 110 requests the data.
  • Flow control is implemented by a simple handshake, so that if host buffer 140 ever were to be depleted, computations of core 110 would suspend until data became available.
  • Host buffer 140 can transfer data from separate address areas for R, G, B, alpha and Z data.
  • the alpha data is optional and Z data may be 2, 3 or 4 bytes long for each pixel.
  • the data for each pixel is transferred to core 110 in a format that matches the data arriving over pixel bus 28 via ASIC 102 , which is usually in the following order: R, G, B, a, Z (a optional).
  • Host buffer 140 is bidirectional or made up of a host input buffer and a host output buffer. As an output buffer, host PCI interface 142 transfers pixel data from core 110 back to the host's address space. Host buffer 140 supports the same modes, buffering and address spaces in both directions, and as an output buffer, host PCI interface 142 splits the pixel data from core 110 into separate streams if required by the host's storage format. Host PCI interface 142 exchanges a handshake with core 110 to confirm that each pixel has been accepted; if host PCI interface 142 withholds this handshake, core 110 will not produce further pixel data until the handshake occurs.
  • control, status, and address registers are all visible on PCI bus 136 , they are also visible across pixel bus 28 to the upstream and downstream merge engines. It is therefore possible for a merge engine to evaluate the state of either of its nearest neighbors by reading the status registers, and to change this state by writing control and/or address registers. This capability is important for implementing high-speed control operations on the pipeline. Such operations are much faster when they are implemented in this way rather than by requiring every host to participate. Performing these operations at high speed (on the order of milliseconds) is important for achieving high performance for multi-pass rendering algorithms that are commonly used for high quality animation and some games.
  • merge engine 24 ( 1 ) The exact set of registers is dependent on the particular implementation, but one exemplary set of registers is shown below.
  • One system design allows the host associated with merge engine 24 ( 1 ) to write to a set of shadow registers in merge engine 24 ( 1 ) and then command merge engine 24 ( 1 ) to copy the shadow register values to a set of shadow registers in merge engine 24 ( 2 ).
  • Merge engine 24 ( 2 ) then determines whether to overwrite its registers with the shadow register values and to overwrite the registers if merge engine 24 ( 2 ) determines that it would be appropriate to do so.
  • Merge engine 24 ( 2 ) can then write the shadow register values to merge engine 24 ( 3 ) and so on.
  • Each merge engine uses an identity mask to decide whether to overwrite its local set of registers with the shadow register values. If the value of a merge engine's local identity mask matches the value of the shadow register identity mask, and then the merge engine accepts the register values, otherwise the merge engine ignores them.
  • Table 1 lists the fields of a status register and Table 2 lists the fields of a control register.
  • the status register and control register are always readable by the merge engine's host.
  • StopRunning 1 Set high to stop merge engine logic 100 from running AssertException 1 Set high to assert exception Reset 1 Set high to reset merge engine ResetFrame 1 Set high to reset current frame Snapshot 1 Set high to snapshot all internal states NoPropagate 1 Set high to prevent propagation of control messages
  • Registers 144 also include a snapshot register, an M-byte register, which holds an M-byte result of a last snapshot request. The snapshot register is readable when the merge engine is stopped. Registers 144 also include an ASIC address register and an ASIC data register used for accessing the state of ASIC 102 . The ASIC address register is an eight-byte register that indicates a target of the data held in the ASIC data register.
  • registers 114 Two other registers in registers 114 are host readable and writeable, but only when the merge engine is stopped. Those two registers are an address register, the fields of which are shown in Table 3, and a control/status register, the fields of which are shown in Table 4. TABLE 3 ADDRESS REGISTER Field Description HostInRgb Base address for reading frame buffer pixels HostInAlpha Base address for reading frame buffer alpha HostInZ Base address for reading Z buffer HostOutRgb Base address for writing frame buffer pixels HostOutAlpha Base address for writing frame buffer alpha HostOutZ Base address for writing Z buffer
  • FIG. 3 An image generation operation using the image generator of FIG. 1 with the merge engine of FIG. 2 will now be described with reference to FIG. 3.
  • the merge engines 24 are shown with direct connections, but it should be understood that these connections might well be via pixel bus 28 .
  • Each stage, or thread, of the parallel image generation process comprises a rendering engine (RE) 22 and a merge engine (ME) 24 .
  • Each stage might be implemented on a different host running different threads of a multithreaded program, as explained below in connection with FIG. 4.
  • each stage processes one shape and those shapes are a circle, two rectangles and a triangle.
  • Image controller 20 passes an object description of the circle to rendering engine 22 ( 1 ) along with a command to initiate rendering of the circle.
  • the object description of a circle includes a location for the center of the circle and a radius in the screen coordinate space, along with a description of the surface of the circle.
  • the description of the circle's surface could be as simple as a solid color, but more often it is a complex description, possibly including a texture or a description of the reflectance/transmission properties assigned to the object and a description of the light sources or shadows impinging on the object.
  • Image controller 20 also passes the three other objects to their respective stages for parallel processing.
  • rendering engine 22 ( 1 ) determines blocks of pixel color values, it outputs those blocks as a pixel stream to merge engine 24 ( 1 ) and eventually outputs all pixel values comprising part image 170 ( 1 ).
  • Rendering engines 22 ( 2 )-( 4 ) do the same, outputting part images 170 ( 2 )-( 4 ) in parallel as they are generated.
  • rendering engines 22 preferably also output Z values for the pixels, where a Z value for a pixel indicates the depth of the portion of object represented by that pixel.
  • Merge engines 24 use the Z values to determine when one object obscures another.
  • each of the objects are placed at constant depths, with objects OBJ 1 -OBJ 4 being at depths of 1, 3, 2 and 4, respectively.
  • each stage is supported by one workstation and various threads of a multiprocessing program are parsed out to the workstations.
  • FIG. 4 shows details of one such system.
  • one workstation controls the overall image generation process by running a program thread (TH 0 ) that communicates with threads in other workstations 180 over a network interconnecting the workstations.
  • API 0 distributes rendering commands from TH 0 among the set of threads TH 1 through TH N 190 running on workstations 1 through N.
  • Each thread TH i relays the rendering commands, with minimal overhead, to an API i 192 on workstation i, which can be an unmodified OpenGL API that drives a single graphics accelerator card GA i 194 .
  • Thread TH 0 accesses the set of N graphics accelerator cards (GA i ) 194 housed in some N workstations (or fewer than N if multiple graphics accelerator cards can be installed in one or more workstations) via application program interface API 0 on workstation 0 that is running TH 0 .
  • API 0 distributes the rendering information to the graphics accelerators by communicating with a set of N threads (TH i ). Each thread invokes a local API to drive a graphics accelerator to produce an image in a frame buffer and this image is provided to the thread's corresponding merge engine.
  • This image is passed over a private bus to a merge engine (e.g., merge engine 24 ( i )) from it is fed into a ServerNet®-based pathway of merge engines.
  • the final merge engine, 24 (N) deposits the fully processed image back into the final frame buffer, or a dedicated frame buffer, from where the image can be sent to a display.
  • One data path of concern is the network path between API 0 and the various threads.
  • a typical graphics accelerator today has a peak throughput of a few million triangles/second, but assuming a realistic utilization of 25%, or 250 , 000 triangles, the bandwidth from API 0 to each thread 190 would be between 2.5 and 5 MBS, assuming that texture data has been preloaded and is locally available to each accelerator card 194 . If multiple accelerators are housed in a single multiprocessor workstation, this bandwidth can be easily supported through the memory system.
  • API 0 is in a separate workstation from the threads, local network links can easily carry the data traffic, over a ServerNet® network bus or some other network, so that data path is not a problem.
  • the application might be designed with explicit parallelism in mind, either as a SPMD (Single Program, Multiple Data) program or MIMD (Multiple Instruction, Multiple Data) program.
  • SPMD Single Program, Multiple Data
  • MIMD Multiple Instruction, Multiple Data
  • thread TH 0 and API 0 cooperate to send high level instructions to threads 190 that are application threads.
  • the model data is distributed in such a way that it is available locally to threads 190 and does not have to be transferred between API 0 and threads 190 .
  • thread TH 0 sends instructions to the other threads 190 to manipulate the model or viewing position and threads 190 perform local updates to the model data and initiate the rendering commands that drive their APIs 192 .
  • the image generation is not limited by the inter-workstation bandwidth, since a model can be, for example, rotated in real-time simply by broadcasting a change of viewing position, which can be defined by 24 bytes.
  • the bandwidth needed between API 0 and threads 190 is only these 24 bytes, since the model itself is located on the workstations that contain the graphics accelerators.
  • the connections among the merge engines could be realized by the ServerNet® links and switches.
  • ServerNet® links and switches could be more expensive than a pipelined configuration it would be a natural way to perform rendering on a cluster computer that was already ServerNet®-connected. It would also be an appropriate way to build a fault-tolerant rendering system, and this could be important for real-time applications that require high reliability, such as flight simulators and location based entertainment.
  • the most desirable configuration would combine a pipelined architecture, using dedicated ServerNet® links, with a general purpose ServerNet® network used for all other communication.
  • the pathway is from merge engine 24 ( 1 ) to merge engine 24 ( 2 ), and so on, to merge engine 24 (N) and then to merge engine 24 ( 1 ).
  • the order of the merge engines in the pathway is not fixed, but is controlled by a switch network 200 .
  • switch network 200 might be a physical message router, while in other cases, switch network 200 is implemented by dynamically modifying the merge engines' addresses on pixel bus 28 or the destinations of their message.
  • One use of a dynamically ordered merge pathway is to allow for noncumulative operations, as might be needed for handling fog and transparency in an image.
  • it may be appropriate to combine pipelined and switched configurations.
  • the pipelined configurations are scalable and inexpensive, while the fault tolerant configuration is appropriate for high reliability applications including flight simulation and location based entertainment.
  • the fault tolerant configuration provides double the link bandwidth of the pipelined configuration because it allows ServerNet® channels to communicate in two directions simultaneously.
  • the merge engines (specifically, the x and y ports of ASIC 102 ) are connected to a ServerNet® switch that is a 300 nanosecond latency crossbar with 15 gigabit bisection bandwidth (in current implementations). Data flows bi-directionally on both the x and y links, with the upstream and downstream traffic split across the two links.
  • the arbiter is responsible for merging the results of an incoming stream that has been split across two links.
  • Image resolutions range from a low of 640 ⁇ 480 pixels, used in video games and current television standards, through 2000 ⁇ 1000 for HDTV imagery, and higher for flight simulators and high-end film animation.
  • Table 5 identifies some target frame rates and resolutions along with corresponding bandwidth requirements. The data requirements vary from 5 to 8 bytes per pixel depending on the resolution of the Z buffer (2, 3 or 4 bytes) and whether an alpha channel is included.
  • bandwidth can be increased by using multiple rendering systems with duplicate model data, with each system rendering a portion of each frame. The results are then combined to make a full frame. Alternately (but perhaps more problematically) each system could render alternating frames in a round-robin sequence.
  • latency should also be considered.
  • the rendering process incurs latency in three stages: first, while application threads prepare data for the graphics accelerators; second, while graphics accelerators generate pixel and Z data for the frame buffers; and third, while the frame buffer data flows through the pathway of merge engines to produce a final image. If the combined latency from these stages is too high (roughly, above 100 ms) then interactive users will experience motion sickness. Since the threads will introduce some latency, and the graphics accelerators may typically introduce additional latency, the merge pathway should nominally operate within a 30 ms time window.
  • the total pathway latency, LT, for an N stage pipeline is:
  • L is the latency of each stage and P is the number of packets or pixels being processed.
  • P is the number of packets or pixels being processed.
  • Latency/pixel Frequency 640 ⁇ 480 97.6 ns 11 MHz 1280 ⁇ 1024 22.9 ns 44 MHz 2000 ⁇ 1000 15.0 ns 67 MHz 1700 ⁇ 1350 13.1 ns 77 MHz
  • Those frequency targets can be achieved in three ways: by adjusting the clock rate of merge engine logic 100 subject to available bandwidth on pixel bus 28 ; by parallelizing the data pathways in merge engine logic 100 (subject again to available pixel bus 28 bandwidth); or by deploying multiple merge pathways in parallel, subject to available bandwidth through the PCI bus and host subsystems.
  • Another approach is to use data compression between the merge engines, as described in an above-referenced co-filed patent application, to increase the effective bandwidth of pixel bus 28 .
  • FIG. 6 is a high-level block diagram of the components that make up one node processor 300 .
  • node processor 300 is hosted by a workstation (not shown) that is connected to other workstations via a ServerNet® network, with logical channels coupling processor node 300 to its two nearest neighbors, a previous workstation and a next workstation.
  • a processor node can be a rendering node, a display node or both.
  • the order of the nodes can be dynamically altered by altering the logical connections between workstations.
  • node processor 300 includes a CPU 302 , coupled via a host bridge 303 to a host memory 304 , a 3D accelerator card 306 and a compositing card 308 .
  • the coupling is provided by a host PCI bus 309 .
  • 3D accelerator 306 is shown comprising a 3D rendering chip 310 and a graphics memory 312 .
  • Compositing card 308 is shown comprising a merge engine chip 320 and a network interface chip 322 (a Colorado ASIC for interfacing to a ServerNet® network).
  • node processor 300 The standard data path within node processor 300 is shown by the numbered arrows. Using its local data or data provided externally, CPU 302 prepares 3D data and stores it in host memory 304 (arrow 1 ). That data is sent, as OpenGL commands, over host PCI bus 309 to 3D accelerator card 306 (arrow 2 ). 3D rendering chip 310 then executes the OpenGL commands and renders 3D objects to graphics memory 312 (arrow 3 ).
  • merge engine 320 In a typical graphics processor, at this point the color data are read out of memory 312 and displayed. However, here the data is sent to the merge engine (arrow 4 ).
  • the preferred way for merge engine 320 to read data from graphics memory 312 is through a private interface as shown in FIG. 6. This method is preferred because it alleviates the host PCI bus from supporting the bandwidth required for this data transfer.
  • Another way for merge engine 320 to read data from graphics memory 312 is to program merge engine 320 to perform direct memory access (DMA) using the PCI interface, rather than having 3D accelerator 306 DMA the data or doing a transfer via a main memory. While this is a good approach if such capability is available, it might not always be available. For example, if and AGP 3D accelerator that implements Intel's AGPSets, PCI-to-AGP DMA reads would not be supported.
  • the best performance of a 3D accelerator card that uses the Evans & Sutherland RealImage 2100 chip set is obtained with bursts of four 32-bit words and a number of wait states between transactions that depends on the direction of the data transfer.
  • the typical data set includes color (RGB), transparency (a) and depth (z).
  • this data set is received in a scan-line fashion simultaneously from the graphics memory 312 and from the upstream node (arrow 5 ).
  • the merge function actually applied to the local data and the upstream data is user-configurable, as explained above.
  • Merge engine 320 applies the merge function in use to the local data and the upstream data and then sends the resulting merged data set to the downstream node (arrow 6 ).
  • Processor node 300 is a Compaq workstation, such as an IA32, or an Alpha processor-based workstation running Microsoft's Windows NT operating system
  • 3D accelerator card 306 is a Compaq PowerStormTM 300 3D accelerator
  • compositing card 308 comprises a Compaq PCI Pamette card (marketed as Compaq's “PCI Development Platform”), version 1, revision 2 , using 4 user-programmable FPGA's (part number XC4044XL-1, marketed by Xilinx).
  • Compositing card 308 also includes a custom Compaq ServerNet® PMC daughter card for the PCI to ServerNet® network interface.
  • each processor node operates on the entire image, but only with a subset of the objects.
  • the merge engines then merge these individual images together.
  • each processor node processes all the objects, but only for a part of the screen attributed to that processor node. In that case, the merge engines are used to tile together the results from the individual processor nodes. The latter approach is useful for parallel ray tracing.
  • the logical topology for the network is the chain, wherein each node has one input data stream and one output data stream, but other topologies can be used, such as a loop, or a tree.
  • a loop, or ring is just a chain with the two chain ends closed to each other.
  • a loop is useful where the data has to be recirculated in the rendering system. While a tree provides lower latency, it requires more network bandwidth, which is often a scarce resource.
  • the nodes are connected, it can be useful to connect all the network inputs and outputs to a switch so that the order of the nodes can be arbitrarily reordered. Dynamic reordering is useful where merging operations require a specific order and that order changes, such as when the viewpoint changes.
  • FIG. 7 shows the components of merge engine 320 in more detail, including a separation of various components according to which of the four FPGA areas 330 ( 1 )-( 4 ) is used for the components.
  • a graphics system using processor nodes 300 can pipeline processing very aggressively. With double buffering for the color and depth buffers, pipelining as shown in FIG. 8 can be obtained. On each node, rendering can start as soon as both the previous rendering and the previous merging are finished. Merging can begin as soon as the previous rendering and the previous merging are finished, but may stall if upstream data is not received or there is no space downstream to store the results (shown in FIG. 8 by cross-hatching). Because of the node-to-node flow control, no broadcast is needed to stop and start the data flow. This way, even on a system with a high node count (such as 128 nodes or more), stalls stay local and the data flow resumes with a latency of one node.
  • a high node count such as 128 nodes or more

Abstract

An image generator is organized into a plurality of rendering engines, each of which renders an image of a part scene and provides the part image to a merge engine associated with that rendering engine. The image is a part image in that it usually contains less than all of the objects in the image to be rendered. The merge engine merges the part image from its associated rendering engine with the part image provided by a prior merge engine and provides the merged part image to a next merge engine. One or more merge engines are designated the output merge engines and these output merge engines output a merged part image that is (a portion of) the ultimate output of the image generator, the full rendered image. Each merge engine performs its merge process on the pixels it has from its rendering engine and from its prior neighbor merge engine, in a pipelined manner and without necessarily waiting for all of the pixels of the part image or the merged part image.

Description

    BACKGROUND OF THE INVENTION
  • The present invention relates to image generation in general and in particular to rendering engines for generating images that might be needed to be generated in real-time response to an input of objects in a scene. [0001]
  • Rendering is the process of computing a two-dimensional viewpoint dependent image of a three-dimensional geometric model. Rendering is computationally intensive, since a typical geometric model might contain objects that collectively comprise millions of polygons. In the typical rendering engine, a geometry stage first converts mathematical descriptions of arbitrarily complex objects into a collection of polygons, typically triangles, so that subsequent rendering stages only need to deal with simply polygons. The rendering stages then need to simulate the transport of light at various wavelengths emitted from numerous light sources in the geometric model and scattered by surfaces of objects in the geometric model. [0002]
  • Two common methods of handing light-object interactions are known as “ray tracing” and “radiosity” methods. If those methods are implemented in software, the software can take from minutes to days to produce a single high quality “photorealistic” image of a large geometric model. A typical photorealistic image might require a resolution of 2000 pixels by 1000 pixels and contain a million or more separately specified scene elements (i.e., objects or light sources). While days might be available for rendering some images, many applications require real-time rendering of video frames, such as computer games, flight simulators, scientific visualizers, computer-aided design/computer-aided engineering (CAD/CAE) applications and medical data visualizers. [0003]
  • With real-time rendering of video frames, the delay allowed between the receipt of a geometric model and the output of the corresponding image is determined, in part, by the frame rate. In some applications, the frame rate can be as low as one frame per minute, so a viable rendering system for that application must produce at least one image per minute if it is to run at an acceptable rate. In many applications, such as computer animation or flight simulation, the rendering system is expected to run at thirty frames or more per second. In some thirty frame/second applications, the delay might need to be even less, as might be the case for a flight simulator or computer game having feedback. With such a system, an operator of the system provides inputs, such as commands to move up, down or sideways, and the corresponding changes to the geometric model are provided to a rendering engine. To avoid the feeling of “sea-sickness” that occurs when an image is out of sync with other stimuli, the rendering engine might have to generate an image from a geometry model in even less time than a frame period. [0004]
  • Existing software rendering engines are generally unable to meet such performance requirements, but some hardware rendering engines have been able to approach these performance requirements, although not always for a practical cost. Generally, in order to generate images of reasonable resolution in {fraction (1/30)}[0005] th of a second or less, a hardware approach known as a “graphics pipeline” is used to achieve real-time image generation.
  • A graphics pipeline comprises a few well-understood stages: the geometry stage, the rendering stage and the composition stage. In a rendering engine with a graphics pipeline, some or all of these stages are implemented in parallel. [0006]
  • In the geometry stage, a database of geometric objects (usually triangles) is read from host memory and transformed from a world coordinate system (“object space”) into a view dependent coordinate system (“screen space”). In some cases, the geometry stage might also perform the conversion of complex objects into simplified objects comprising polygons. Because all polygons can be represented as triangles, and processing triangles is simpler than processing polygons with more than three sides, nearly all existing rendering engines limit geometric objects to triangles. With triangles, each vertex is defined by a coordinate (x, y, z) and a surface normal vector (dx, dy, dz) and might also be defined by surface material properties (such as a coefficient of reflectivity and transmissivity) and the surface may be bound to textures. The geometry stage converts the vertex coordinates and normals from object space representations into their screen space counterparts (with a resolution being a function of the resolution of the output image) and discards objects that fall entirely outside the perimeter of the screen (the “view surface”). The geometry stage also performs lighting calculations, as needed, at the vertices for use by later stages that will interpolate these results at individual pixels. [0007]
  • Once the geometric model is represented as polygons in screen space, the rendering stage calculates the pixel color values that correspond to those polygons. Where the rendering stage is performed by a plurality of rendering engines, the composition stage combines the results from the parallel rendering engines to form the final image. With parallel processing, there are several considerations, such as load balancing and thread independence. The goal of load balancing is to ensure that the work to be done is evenly distributed over all of the parallel threads, to avoid having some threads idle while others are still processing. Thread independence is desirable, since independent threads will not be held up waiting for other threads to reach a certain point in their processing. Some thread dependence might be unavoidable, but such dependence might not always result in additional delays. [0008]
  • One approach to dividing up the work among a plurality of threads is known as the “sort-middle” architecture. In a “sort-middle” approach, the screen space is divided into tiles and each tile is assigned a bin, where the bin corresponds to the work one rendering thread will perform. The objects comprising the geometric stage's output are then allocated to the bins, with each object only being allocated to the bins of the tiles that the object intersects. Within each bin, a Z-comparison (depth comparison) is performed to determine which object is closest to the screen within each pixel of that bin's tile. Shading and blending calculations are then performed to compute the amount of light reflected at each wavelength (typically the three wavelengths r, g and b) by the closest object in each pixel, after textures have been taken into account. The result of this calculation is the color value (e.g., r, g and b values) of the pixel. [0009]
  • Another approach is the “sort-last” architecture wherein, instead of subdividing the screen, the objects are allocated randomly to the bins and each thread does a Z-comparison on the objects in its bin. Each thread computes an entire screen image, albeit an incorrect one containing less than all of the objects in the geometric model. The Z (depth) information is stored along with the pixel color values. [0010]
  • The composition stage in a sort-middle architecture simply combines the subimages for each tile into the final image. The composition stage in a “sort-last” architecture uses the Z information to merge all of the part (“incorrect”) images to produce a single correct image. With a sort-last composition stage, a merge engine is used to combine the part images by identifying which objects in which part images overlap which objects in other part images. In some cases, where there are many parallel threads, the composition stage might be done in parallel. Sort-last architectures typically provide better load balancing than sort-middle architectures, since some sort-middle threads might be allocated tiles that contain few or no objects. [0011]
  • The actual process of rendering, in a single thread, using hardware is highly developed and many products are available that can quickly render objects into images. Such products often support standard command sets, such as the OpenGL API (application programming interface), a low-level programming interface that is closely related to the architecture of the graphics pipeline. This standardization makes those products convenient for development of rendering engines. Another command set that is becoming a standard is the Direct3D API developed by Microsoft Corporation. Direct3D is a higher level programming interface that has been used on computers primarily for games. [0012]
  • Inexpensive rendering cards for personal computers are able to render about one million triangles per second. However, if a thirty frame/second video image is needed, the most complicated model that the rendering card could process in real-time would be limited to no more than about 33,000 triangles. Some computer games might be playable with such a limited number of triangles, but generating a photorealistic image with 33,000 triangles is difficult or impossible. Many CAD/CAE applications, such as those used by auto makers require finite element models with a hundred thousand to a million elements. To render an automotive model with a hundred thousand to a million triangular elements in real-time at thirty frames per second, a rendering engine would need to process three to thirty million triangles per second. A model with five million elements, which may be realistic within a few years, would require a rendering engine that can process 150 million triangles per second. [0013]
  • The bandwidth required at the input to the geometry stage of the graphics pipeline can be estimated from the model size. If a rendering card can process a large number of triangles/second, it cannot be used to its full capacity unless the communication channel to that rendering card can carry all of the data specifying those large numbers of triangles. [0014]
  • Each triangle is represented by at least three vertices, and each vertex by six floating point numbers (a coordinate and a surface normal vector) for a total of seventy-two bytes in the typical specification of a triangle. Surface normal vectors are usually defined at each vertex when the triangle represents a facet of a tessellated surface such as a spherical object. When this is not the case, the data requirements can be reduced by defining a common surface normal vector for all vertices. Another data reduction method is to use “triangle strips” which eliminate redundant vertex definitions, and require one additional vertex for each additional triangle. Worst case, a rendering card that can process one million triangles per second would require a 72 megabyte/second (MBS) communication bandwidth. Many rendering cards for personal computers communicate over a standard peripheral bus known as the PCI bus, and 72 MBS is about the achievable capacity of the current PCI bus. A system that renders three million triangles per second requires a bandwidth of roughly 216 MBS. This bandwidth should be achievable over newer buses, such as Intel's Accelerated Graphics Port (AGP) bus. [0015]
  • Even if a rendering system could render 30 to 150 million triangles/second, it would require a communication channel that could handle from two to eleven gigabytes/second (GBS), which most certainly would require parallel threads for communication. While several high-performance rendering engines that use parallel processing are available, they are often quite costly and not scalable. [0016]
  • SUMMARY OF THE INVENTION
  • The present invention overcomes several disadvantages of the prior art methods and apparatus for generating images. [0017]
  • In one embodiment of an image generator according to the present invention, the image generator is organized into a plurality of rendering engines, each of which renders an image of a part scene and provides the part image to a merge engine associated with that rendering engine. The image is a part image in that it usually contains less than all of the objects in the image to be rendered. The merge engine merges the part image from its associated rendering engine with the part image provided by a prior merge engine (the prior neighbor in a merge engine sequence) and provides the merged part image to a next merge engine (the next neighbor in a merge engine sequence). One or more merge engines are designated as output merge engines and these output merge engines output a merged part image that is (a portion of) the ultimate output of the image generator, the full rendered image. Each merge engine performs its merge process on the pixels it has from its rendering engine and from its prior neighbor merge engine, without necessarily waiting for all of the pixels of the part image or the merged part image. [0018]
  • One advantage of the image generator is that the operations of the merge engines are pipelined, since they operate on pixels as they arrive, if an operation can be done on the arrived pixels, instead of waiting for all pixels to arrive. [0019]
  • In one variation of the basic image generator, the merge sequence is dynamic. In one such embodiment, the merge engines are coupled to a switch network where the switch network determines which merge engines are prior neighbors and next neighbors to which merge engines. Preferably, the switch network makes those determinations based on the objects being provided to the individual rendering engines. [0020]
  • The image generator can be used for a variety of image compositing and “two-and-a-half-D” applications, which combine 3D rendering, 2D overlays and special operations including color correction, dissolves, blending and other transformations. [0021]
  • A further understanding of the nature and advantages of the inventions herein may be realized by reference to the remaining portions of the specification and the attached drawings.[0022]
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a block diagram of an image generator according to one embodiment of the present invention. [0023]
  • FIG. 2 is a block diagram of a merge engine as might be used in the image generator of FIG. 1. [0024]
  • FIG. 3 is a block diagram illustrating an image composition process according to the present invention. [0025]
  • FIG. 4 is a block diagram of a parallel image generator implemented with multiple workstations. [0026]
  • FIG. 5 is a block diagram illustrating the use of a switch network. [0027]
  • FIG. 6 is a block diagram of a specific implementation of a merge engine node in a parallel pipeline graphics processor. [0028]
  • FIG. 7 is a more detailed block diagram of the merge engine shown in FIG. 6. [0029]
  • FIG. 8 is a timing diagram illustrating the pipelining of the graphics process performed by the processor shown in FIGS. 6-7. [0030]
  • DESCRIPTION OF THE PREFERRED EMBODIMENTS
  • The present invention has many applications, as will be apparent after reading this disclosure. In describing an embodiment of an image generator according to the present invention, only a few of the possible variations are described. Other applications and variations will be apparent to one of ordinary skill in the art, so the invention should not be construed as narrowly as the examples, but rather in accordance with the appended claims. [0031]
  • The construction and operation of a basic image generator according to one embodiment of the present invention is shown in FIG. 1. There, a [0032] graphics system 10 is shown with an image generator 12 as might be used to render an image from an object description file 14 and present a rendered set of pixel color values to a display 16 or an image store 18. Object description file 14 can be any type of data file or stream of expressions that indicate where objects and light sources are in a scene, as well as other parameters needed to render an image according to conventional methods. While only two output devices are shown, the output of image generator 12 might be used in other ways, such as being fed into another computer system or program for further analysis and/or processing of the rendered image, or any other use typically made of rendered images.
  • The image generator itself is shown comprising an [0033] image controller 20, a plurality of rendering engines 22, a plurality of merge engines 24 and a frame store 26. In many implementations a paired set of a rendering engine and an associated merge engine might be implemented on one workstation, as described below in reference to later figures, but this description of the basic system is not limited to any particular limitation on the allocation of the physical hardware. Herein, when a specific rendering engine or merge engine (or any other individual from a plurality), an index is indicated, as in “22(i)” or “22 i” to identify the i-th rendering engine. In FIG. 1, rendering engines 22(1), 22(2), 22(N−1) and 22(N) are shown and, for N>4, rendering engines 22(3) through 22(N−2) are implied.
  • The data paths and connections shown in FIG. 1 provide for an input of an object description file to image [0034] controller 20 and for transfer of subsets of the objects to rendering engines 22, e.g., object(s) OBJ1 to image controller 22(1), object(s) OBJ2 to image controller 22(2), etc., in a “sort-last” or “sort-middle” manner. In some embodiments, image controller 20 might be configured to send only one object to each rendering engine 22, i.e., the description OBJi is the description of just one object. However, in other embodiments, image controller 20 might send multiple objects to one or more rendering engine, to load balance (i.e., give each rendering engine roughly the same amount of work to do) or to distribute all of the objects at once when there are more than N objects in the scene. Image controller 20 is shown in FIG. 1 as one unit, but in some embodiments, image controller 20 is implemented as a distributed application. In those embodiments, the application might be distributed over a collection of machines that are the same machines that are hosting the rendering engines 22 and merge engines 24. If that is the case, then host bus 32 might be replaced with a set of host busses, one per host, to connect the host component of the image controller to its corresponding rendering engine and merge engine.
  • A data path is provided between each [0035] rendering engine 22 and its corresponding merge engine 24 for communicating a part image from the rendering engine to the merge engine, wherein the part image PIi communicated from rendering engine 22(i) to merge engine 24(i) is a rendered image of the object(s) described by OBJi. Each of the merge engines 24 is also provided with a data path to its nearest prior neighbor and its nearest next neighbor. Thus, for merge engine 24(i), a data path is provided from merge engine 24(i−1) and a data path is provided to merge engine 24(i+1). Merge engines are logically arranged in a ring, so that the nearest prior neighbor to merge engine 24(1) is merge engine 24(N) and the nearest next neighbor to merge engine 24(N) is 24(1). It should be understood that, where N elements (numbered from 1 to N) are described herein as being in a ring and the i-th instance of an element is referenced, “i−1” and “i+1” refer to the i-th elements' nearest neighbor, and because of the ring arrangement, for i=1, “i−1” refers to the N-th element and, for i=N, “i+1” refers to the first element.
  • The data paths between merge engine neighbors can be direct connections, but the embodiment shown in FIG. 1 uses a high-[0036] speed pixel bus 28 to carry the traffic between merge engines. One suitable high-speed bus is the ServerNet® network developed by Tandem Computers Incorporated. The ServerNet® network is described in Heirich, Garcia, Knowles and Horst, “ServerNet-II: A Commodity Interconnect for Scalable High Performance Cluster Computing”, which is incorporated by reference herein for all purposes.
  • One of the [0037] merge engines 24 is designated as the output merge engine. In FIG. 1, the designated merge engine is merge engine 24(N), which is provided with a data path to a frame buffer 26 and holds the output of image generator 12, which is a merged image of all of the part images PI. In some implementations, where the rendering engines already include suitable frame buffers, the frame buffer of rendering engine 24(N) is used instead of a dedicated frame buffer such as frame buffer 26. Some implementations might also use more than one output merge engine. Other connections might be provided but not shown as described herein.
  • As described below, one advantage of the arrangement shown in FIG. 1 is that the merger of part images is done in a pipeline, with pixel values forming the part images and merge images streaming from the rendering engines and merge engines as they become available. Thus, as one merge engine completes the merger of a block of pixels streamed from its corresponding rendering engine with a block of pixels streamed from the merge engine's nearest prior neighbor, it passes the merged block on to its nearest next neighbor. The block of pixels can range in size from one pixel to the number of pixels in the image, but as the block size approaches the number of pixels in the image, the benefits of pipelining diminish, and are not obtained when the block size is the size of the image. In one prototype, the image size is 307,200 pixels (640 by 480 pixels) and the block size is 8 pixels. [0038]
  • The link between 24(N) and [0039] 24(1) is omitted in some configurations that do not require recirculation of intermediate image results, depending on the nature of the rendering process being employed. However, in general, the link is included.
  • In [0040] image generator 12, each merge engine 24 acquires image data from its corresponding rendering engine 22 through a render-merge interface 30. Where rendering engines 22 are standard graphics accelerator cards with PCI interfaces, render-merge interface 30 can be a PCI interface, but a private digital interface that performs better than PCI interfaces might be preferred. Each merge engine 24 receives pixel color values, and other data, from its corresponding rendering engine 22 and also receives image data from its nearest prior neighbor via high-speed pixel bus 28. Where pixel bus 28 is implemented using a ServerNet® network, the interface to pixel bus 28 could be implemented using a ServerNet® “Colorado” application-specific integrated circuit (ASIC) developed by Tandem Computers Incorporated for handling communications over a ServerNet® network. A merge engine 24(i) combines its two streams of image data to produce a third stream, which is sent through an outgoing connection on ASIC to the next merge engine 24(i+1) in the pathway. At one or more points in the pathway, a merge engine will push the combined imagery onto a frame buffer to allow for display of the image. The ASIC is documented in detail in the “Colorado External Specification”, Tandem Computers Document Number 122219-X05 dated Oct. 14, 1997, which is incorporated by reference herein for all purposes.
  • [0041] Image generator 12 might advantageously be built from readily available graphics components, to reduce the design efforts and costs of an image generator. For example, as explained above, rendering engines 22 could be standard graphics accelerator cards and merge engines 24 could incorporate an existing driver for a ServerNet® network interface, such as the Colorado ASIC.
  • A typical, off-the-shelf commodity graphics card has a PCI bus interface (a standardized bus used for inter-board communications in desktop computers) for receiving object descriptions and can render a few million triangles per second (objects are typically broken down into collections of triangles for rendering). An image generator is usually “parallelized” to overcome limits in of processing speed in each thread or limits in bandwidth into, out of, or internal to, the thread. Where processing speed is the stricter constraint, the number of “rendering threads” (“N” in the examples described herein) is determined by dividing the maximum number of triangles/second needed in the application by the number of triangles/second processable by one thread (which is often limited to the number of triangles/second processed by the rendering card that forms part of the thread). Where bandwidth at the input of the rendering stage is the stricter constraint, N would need to be at least the overall bandwidth of the geometry stage output divided by the bandwidth of one card's input. In two specific prototypes, N=6 and N=8. [0042]
  • In some cases, the output bandwidth of the rendering stages is a constraint. Typically, each rendering engine will produce a stream of thirty frames/second of video. Each pixel on the output might comprise 24 bits of color information (8 bits each for red, green and blue), 16 to 32 bits of Z (depth) information, and possibly an additional eight bits of alpha (transparency) data, totaling from 40 to 60 bits of data. In high-end motion picture applications, 12 bits/color might be the norm, resulting in a minimum of 52 bits. For NTSC television video, the typical format is 30 frames/second with each frame comprising 640 pixels by 480 pixels, for a total of 307,200 pixels per frame. Thus NTSC rendering engines would output between 44 and 70 MBS of data. In the near future, television will expand to the HDTV format, with its approximately 2000×1000 pixel frames, requiring a bandwidth of between 286 and 455 megabytes/second. [0043]
  • FIG. 2 is a block diagram showing one merge engine [0044] 24(i) in further detail. Merge engine 24(i) is logically arranged into merge engine logic 100 and a merge engine interface circuit 102 for connecting neighboring merge engines and merge engine interface circuit 102 uses an ASIC for handling the communication between merge engines over a network such as a ServerNet® network. Other connection approaches could be used, but using an ASIC that is used for other applications has well-known advantages, such as the fact that it eliminates the need to design communications logic specifically for the merge engine communication and having a broader base of uses for any given ASIC distributes the design costs over more chips, thus reducing the per-chip design costs. In combination with the use of off-the-shelf components for the rendering engines and the use of an already designed ASIC for communications, the N pipeline stages of an image generator can be implemented very economically.
  • Three interfaces are shown in FIG. 2. At the top, an [0045] ASIC 102 connects the merge engine to its nearest neighbors over pixel bus 28. The other two interfaces are a part image bus 30 between the merge engine and its corresponding rendering engine and a host PCI bus 32 between the merge engine and a host which is used for device initialization and host control.
  • The connections to [0046] pixel bus 28 are labeled “x” and “y”, as those are the two channels provided by the Colorado ASIC. The Colorado ASIC makes the design of the merge engine shown herein quite cost effective, especially since the ASICs can be used essentially unchanged. Each ASIC has two ports (termed “x” and “y”) that are usually used redundantly to provide fault tolerance. In a basic embodiment, the “x” and “y” points are used for data flow in one direction, but in other embodiments, those parts, which are bi-directional, are used to carry data in both directions, giving higher effective bandwidth and frame rate.
  • [0047] Merge engine logic 100 comprises various elements for communication between a function-unit core (core 110) 110 and the three interfaces. Core 110 operates on pixel streams arriving at the merge engine over pixel bus 28 (from other merge engines) and over part image bus 30 from an associated rendering engine. In particular, core 110 performs operations on pairs of pixels.
  • [0048] Core 110 is connected to ASIC 102 through a PCI interface unit 120. Incoming pixel data flows from PCI interface unit 120 through an arbiter 122, an input buffer 124 and a decoder 126, while outgoing pixel data flows from core 110 to PCI interface unit 120 through an encoder 130, an output buffer 132 and a sender 134.
  • When image data is transferred from one merge engine to its nearest next neighbor, the image data is transferred as a message of a fixed multiple of the ServerNet® network packet data size. When a message arrives on the input to [0049] ASIC 102, ASIC 102 passes the message into input buffer 124 via PCI interface unit 120 and arbiter 122. The PCI bus 136 between ASIC 102 and merge engine logic 100 is not required, as other bus protocols might be used, but with a standard PCI interface to merge engine logic 100, the existing interface of the Colorado ASIC need not be redesigned.
  • At the same time that the message arrives from [0050] ASIC 102, another message of the same length arrives over the part image bus 30 into a host buffer 140. Core logic handles flow control, packet synchronization, and data transfers between the merge engine and rendering engine. With the arrival of these two messages, core 110 can perform pixel-by-pixel computations on these two messages to produce a result message with the same fixed length. The details of the possible computations are described below with reference to core 110 operation. The result message is sent to ASIC 102 for transmission to the merge engine 24(i+1).
  • Merge engine [0051] 24(i) usually operates on two input streams of pixel data: one from merge engine 24(i−1) and one from rendering engine 22(i) and usually produces a single output stream of pixel data, which is sent downstream to the merge engine 24(i+1). In some cases, the merge engine may use only a single input stream, may obtain additional data from a host, and/or may write its output stream to the host. These operating modes are defined by control registers in the merge engine that may be set independently by the host or by control tokens arriving over pixel bus 28.
  • Input pixel data is written into [0052] input buffer 124, which is a circular buffer, unless access to input buffer 124 is denied by the arbiter 122. Decoder 126 removes data from input buffer 124 and decompresses the data, one pixel at a time, at the request of core 110. In some implementations, compression is not used and decoder 126 and encoder 130 are not needed. Decoder 126 informs arbiter 122 whether input buffer 124 has room to accept new incoming data and arbiter 122 uses this information to decide when to deny access to input buffer 124.
  • [0053] Host buffer 140 unit supplies data to core 110, one pixel at a time, at the request of the core 110. Host buffer 140 contains data from up to four separate segments of host address space, in blocks that are sufficiently large to ensure that data is always available when requested by core 110. Host buffer 140 is bi-directional, in that it will also buffer data being sent from core 110.
  • In the normal course of operations, [0054] core 110 requests data for one pixel from decoder 126 and one pixel from host buffer 140. It uses that data to compute values for one new pixel, which core 110 then provides to encoder 130. Encoder 130, when requested by the core 110, compresses the data for a pixel and inserts the result into output buffer 132, which is also a circular buffer. Encoder 130 informs sender 134 when output buffer 132 contains enough data to send downstream. When this happens, sender 134 generates the appropriate PCI transactions to move the data from output buffer 132 to the input buffer 124 of merge engine 24(i+1) via ASIC 102.
  • The first merge engine in the pipeline, merge engine [0055] 24(1), usually reads an input stream of pixel data and passes it downstream to merge engine 24(2) without accepting any pixel data from the upstream direction. In some situations, such as a redistribution of Z buffer data, a merge engine will read an input stream from its nearest prior neighbor without taking any data from its rendering engine. In the normal course of operations, the final merge engine 24(N) will write an output stream of pixel data to memory on a frame buffer of rendering engine 24(N) or frame buffer 26, but will not send data downstream. During Z buffer redistribution, every merge engine except 24(N) will write an output stream of Z information to the host rendering engine and also to its next nearest neighbor.
  • In order to support high-speed multi-pass processing, control messages might be sent along the merge pathway to modify status, control and address registers within the merge engines. These registers are shown in FIG. 2 as [0056] registers 144 and are described in further detail below. Each merge engine is assigned a group identifier, GID, and every control message contains a GID. When a merge engine receives a message over pixel bus 28, it compares its GID to the GID found in the message and processes the messages addressed to it, possibly changing some contents of registers 144 in the process. In either case, the merge engine propagates the control message downstream, unless a “do not propagate” control bit in the message is set.
  • [0057] Arbiter 122 implements flow control for input buffer 124, telling decoder 126 when fresh data is available, and preventing the upstream sender 134 from overwriting active data that has not yet been decoded. Arbiter 122 is also responsible for recognizing control messages when they arrive and routing them to the appropriate logic.
  • [0058] Input buffer 124 is a circular buffer that holds pixel data that is waiting to be decoded by decoder 126 (if compression is used) and consumed by the core 110. Core 110 will most likely consume data more quickly than pixel bus 28 can deliver it, so there is little chance of overflow in normal operation and input buffer 124 can be a simple double buffer. It will accept a data and address interface from arbiter 122 and export an interface to decoder 126 that will provide one byte at a time in FIFO order.
  • Decoder [0059] 126 unit decompresses pixel data (if compression is used). One suitable compression scheme is that described in U.S. Pat. No. ______ (U.S. patent application No. ______ filed on the same day as the instant application with attorney docket number 10577-459 or 010577-045900US). If compression is used, decoder 126 decompresses data from input buffer 124, one byte at a time, until it accumulates the data for an entire pixel (usually between 5 and 8 decompressed bytes). Decoder 126 then passes the decompressed result to core 110 when core 110 requests a pixel. Flow control between decoder 126 and core 110 is implemented by a handshake.
  • [0060] Core 110 is the computational heart of merge engine 24. core 110 takes as input the data for a single pixel from decoder 126 and data for a single pixel from host buffer 140. Under some circumstances, core 110 may take additional data from host buffer 140. Core 110 produces result data for a single pixel, which may be sent to encoder 130, host buffer 140 or both. Under some circumstances, the result data might be combined with additional data from host buffer 140 in a two-stage computation. Flow control is implemented by handshakes between decoder 126 and core 110, host buffer 140 and core 110, core 110 and encoder 130, and core 110 and host buffer 140.
  • [0061] Core 110 should not compute the next pixel until it receives handshakes from decoder 126 and host buffer 140 (if two input streams are active) to confirm the data is ready; after core 110 sends the data for the new pixel to encoder 130, core 110 should not generate more pixel data until it receives acknowledgments from encoder 130 (and host buffer 140, if it is an active destination).
  • The functions of [0062] core 110 will now be described. At least two computations should be supported: a Z comparison operation, in which one of the two input pixels is discarded based on their relative Z values, and a blend operation, in which transparent pixels are blended using a variety of blending formulas. Other pixel operations can easily be added to support commercially important markets, including image compositing and digital special effects.
  • [0063] Core 110 processes data that corresponds to image pixels and Z (depth) information. A pixel is usually defined as 24 bits, 8 bits each for R, G, and B color data, or as 32 bits, with an additional 8 bits of “alpha” transparency data. The Z information for a pixel is usually defined by a 16, 24, or 32-bit integer. In a specific implementation, the design of core 110 should not preclude supporting higher numbers of bits in either pixel or Z data. Although the computations of merge engine logic 100 are defined in terms of individual pixels, it should be understood that the computation would proceed most efficiently when blocks of pixels are processed together.
  • [0064] Core 110 will perform different computations in different operating modes. In some circumstances, core 110 may not even perform computation on the data, but may pass the data through unchanged.
  • One operation is a Z comparison, which is expected to comprise nearly 95% of the operations performed by a core. for this operation, given two pixels P1 (R1, G1, B1, Z1) and P2 (R2, G2, B2, Z2), if (Z1<Z2) then the result pixel is P1, otherwise the result pixel is P2. [0065]
  • A more complex operation is “blending” to handle atmospheric effects and transparency. [0066]
  • Alpha blending is used to render transparent surfaces and is the basis for volume rendering. Instead of computing a pixel color from a single polygon, a rendering system can treat polygons as semitransparent objects (for example, the windshield of an automobile) and compute pixel colors by blending the colors of a number of polygons along a single line of sight. Each polygon has an associated “alpha” coefficient, between zero and one, that determines its relative transparency. The blending operation is not commutative, which means that it must be computed in order along a line of sight. This poses some challenges for any parallel rendering architecture because it means that the relative positions of objects must be maintained through the pipeline. [0067]
  • For a blending operation, given two pixels P1 (R1, G1, B1, A1, Z1) and P2 (R2, G2, B2, A2, Z2), with alpha values A1 and A2 being between 0 and 1, the result pixel, PR, may be computed as: [0068]
  • PR=(F(A1,R1,R2),F(A1,G1,G2),f(A1,A1,A1,A2), MIN (Z1Z2)),
  • where F(a,b,c):=a*b+(1−a)*c [0069]
  • Other operations could be defined in order to support digital effects and image processing, preferably limited to operations on a pair of pixels that produce a pixel result. Some examples include pixel difference, minmax, dissolve, color filters, and the like. The data for each pixel might also include data on the nature of the computation to perform on that pixel, as well as the data to be used in that computation, to provide an advantage over systems that cannot specify this on a per-pixel basis. [0070]
  • Other operations might involve moving data, synchronization or signaling. For example, a data pass-through operation might be provided, wherein a merge engine passes a stream of data from either input (host PCI or ASIC) to its output, unmodified. Merge engine [0071] 24(1) will often have to do this in order to feed the head of a merge pipeline. Preferably, pixel data or Z data can be passed independently and from either input.
  • Another operation is data broadcast, as might be used in connection with a multi-pass algorithm to render transparent surfaces, it will be necessary to broadcast data from one frame buffer to some or all other frame buffers. For example, a model containing transparent components can be rendered by first rendering all of the opaque components, merging the results, and then reloading all of the Z buffers with the merged Z result before proceeding to render the transparent components. The pixel data and/or the Z buffer data can be written to the host from data arriving over the ASIC input. Another operation, typically performed by merge engine [0072] 24(N), is the write out of results to the final frame buffer. This operation might be useful in other stages of the pipeline, or for other operations that would require circulating data multiple times between a merge engine and its host.
  • If the merge engine is created using FPGA's as described above, the operations performed by the merge engine can be updated as needed by reprogramming the FPGA's. One implementation of a merge engine uses a custom-designed board that plugs into a PCI slot of a workstation, where the merge engine board contains a ServerNet® Colorado ASIC and commodity and custom circuitry to perform pixel compositing, blending and other operations, as well as to transfer data in and out of workstation memory. The merge engines are connected in series into a (logical or physical) ring topology. In a simple, cost-effective configuration, these boards are connected in a daisy-chained ring among a set of workstations. In more complex configuration, the boards are connected to a switch fabric to increase throughput, provide fault tolerance or to support specialized applications like volume visualization. [0073]
  • The above operations, and other merge engine capabilities, describe mutually exclusive modes in which the merge engines may operate. A mechanism exists for configuring these modes at run time. In the simplest case, every host is responsible for writing a register in merge engine (see the description of the register set below) that determines the current operating mode of that merge engine. An alternative mechanism that would be faster uses the merge engines to pass control tokens through [0074] pixel bus 28. Cores 110 would then reconfigure themselves in response to receiving one of these control tokens. An entire pipeline could be configured in a single operating mode simply by having one host instruct a merge engine to place a control token into the data flowing through the pipeline. In that case, it would be useful to assign group identifiers to the merge engines, so that control tokens can be destined for subsets of merge engines without necessarily affecting all merge engines.
  • An instance of this problem arises with respect to pipeline synchronization. In order to achieve low pipeline latency, the pipeline stages should be overlapped as much as possible. The set of graphics accelerators in the [0075] rendering engines 22 will usually show some variation in the amount of real time they require in order to produce their pixel and Z data. The merge pathway cannot operate until all graphics accelerators have produced their data, and therefore barrier synchronization is required between the rendering and merge operations. Note that rendering can still overlap merging, since the frame buffers are typically double-buffered and the graphics accelerator can start work on the next image while the merge engines are working on the current image. Barrier synchronization accomplished in software will take milliseconds and thus is clearly unacceptable. A distributed synchronization algorithm is implemented in merge engines in order to initiate merging for each image. The following simple process is used in one embodiment by each merge engine MEi:
  • If (MEi+1 asserts “ready” and frame buffer FBi has data) then [0076]
  • If (i>1) then [assert “ready”] else [start the data flowing into the pipeline][0077]
  • Referring now to [0078] encoder 130, that element accepts one pixel of data from core 110, compresses it, and writes the compressed result to output buffer 132. Encoder 130 exports a flag to sender 134 to request transmission of the contents of output buffer 132 downstream to the input buffer of the next merge engine. Encoder 130 also accepts a flag from sender 134 that indicates which half of output buffer 132 encoder 130 should write into. If encoder 130 recognizes that it is about to overwrite data that has not yet been sent downstream (which should only occur if sender 134 is blocked from sending downstream) then encoder 130 will not execute the handshake with core 110 and thus will effectively block further computation in core 110. Otherwise, encoder 130 sends an acknowledgment to core 110 when encoder 130 has encoded the pixel data and written it into output buffer 132.
  • [0079] Output buffer 132 is a double buffer similar to input buffer 124. The contents of output buffer 132 are accessible to sender 134 so that sender 134 can generate the PCI transactions required to transfer the contents to the input buffer of the next merge engine.
  • [0080] Sender 134 is a data movement mechanism, such as a DMA engine, that pushes data, one packet at a time, from output buffer 132 to PCI bus 136 between sender 134 and ASIC 102. Since sender 134 uses normal PCI arbitration to gain access to PCI bus 136, the flow control mechanisms described previously for arbiter 122 will ensure that sender 134 does not overwrite active downstream data. Flow control in the reverse direction is accomplished by exporting a bit flag to tell encoder 130 which half of output buffer 132 to write into. In the switched configurations described below, sender 134 is responsible for splitting the outgoing data stream across two links.
  • A set of control, status and address registers are provided in each [0081] merge engine 24. The registers are always be accessible to the host through host PCI bus 32, although the design of merge engine 24 might include a constraint that a host is not allowed, or expected, to modify some of the registers while computations are occurring in core 110. The registers are also accessible to other merge engines, through the use of control messages passed from merge engine to merge engine over pixel bus 28. If pixel bus 28 is a ServerNet® network bus, this feature can be used to implement a variety of control operations at high speed without involving the individual hosts. The registers are logically arranged in a PCI address space, so they can be read by an adjacent merge engine via ASIC 102 using standard PCI commands.
  • [0082] Host buffer 140 transfers pixel data from up to four independent segments of host PCI address space, buffers it, and assembles it into pixel format for use by core 110. Host buffer 140, or host PCI interface 142 maintains and increments appropriate address registers so that the transfers can occur in bursts in a predetermined order. The data is transferred to core 110 when core 110 requests the data. Flow control is implemented by a simple handshake, so that if host buffer 140 ever were to be depleted, computations of core 110 would suspend until data became available. Host buffer 140 can transfer data from separate address areas for R, G, B, alpha and Z data. The alpha data is optional and Z data may be 2, 3 or 4 bytes long for each pixel. The data for each pixel is transferred to core 110 in a format that matches the data arriving over pixel bus 28 via ASIC 102, which is usually in the following order: R, G, B, a, Z (a optional).
  • [0083] Host buffer 140 is bidirectional or made up of a host input buffer and a host output buffer. As an output buffer, host PCI interface 142 transfers pixel data from core 110 back to the host's address space. Host buffer 140 supports the same modes, buffering and address spaces in both directions, and as an output buffer, host PCI interface 142 splits the pixel data from core 110 into separate streams if required by the host's storage format. Host PCI interface 142 exchanges a handshake with core 110 to confirm that each pixel has been accepted; if host PCI interface 142 withholds this handshake, core 110 will not produce further pixel data until the handshake occurs.
  • Since the control, status, and address registers are all visible on [0084] PCI bus 136, they are also visible across pixel bus 28 to the upstream and downstream merge engines. It is therefore possible for a merge engine to evaluate the state of either of its nearest neighbors by reading the status registers, and to change this state by writing control and/or address registers. This capability is important for implementing high-speed control operations on the pipeline. Such operations are much faster when they are implemented in this way rather than by requiring every host to participate. Performing these operations at high speed (on the order of milliseconds) is important for achieving high performance for multi-pass rendering algorithms that are commonly used for high quality animation and some games.
  • The exact set of registers is dependent on the particular implementation, but one exemplary set of registers is shown below. One system design allows the host associated with merge engine [0085] 24(1) to write to a set of shadow registers in merge engine 24(1) and then command merge engine 24(1) to copy the shadow register values to a set of shadow registers in merge engine 24(2). Merge engine 24(2) then determines whether to overwrite its registers with the shadow register values and to overwrite the registers if merge engine 24(2) determines that it would be appropriate to do so. Merge engine 24(2) can then write the shadow register values to merge engine 24(3) and so on. Each merge engine uses an identity mask to decide whether to overwrite its local set of registers with the shadow register values. If the value of a merge engine's local identity mask matches the value of the shadow register identity mask, and then the merge engine accepts the register values, otherwise the merge engine ignores them. This allows different merge engines to be configured differently. For example, merge engine 24(1) and 24(N) are nearly always configured differently from the other merge engines.
  • The following tables describe the structure of various registers that might make up [0086] registers 144 of a merge engine and how their contents are used. Table 1 lists the fields of a status register and Table 2 lists the fields of a control register. The status register and control register are always readable by the merge engine's host.
    TABLE 1
    STATUS REGISTER
    # of
    Field bits Values Description
    IsRunning
    1 false, true
    IsException
    1 false, true
    IsReset
    1 false, true
    IsResetFrame
    1 false, true
    GroupId (GID) 8   0-255
    IOMode 4 0000-1111 Inclusive OR of SnetIn,
    SnetOut, HostIn, HostOut
    PixelFormat
    4 0000-1111 Inclusive OR of (R, G, B, alpha,
    Z, reserved)
    RgbLength 2 00 = 16 bits Length of RGB pixel data in bits
    01 = 24 bits
    10 = reserved
    11 = reserved
    AlphaLength 2 00 = 8 bits Length of alpha pixel data in bits
    01 = reserved
    10 = reserved
    11 = reserved
    Zlength 3   0-7 Length of Z pixel data in bytes,
    minus two
  • [0087]
    TABLE 2
    CONTROL REGISTER
    Field # of bits Description
    StopRunning
    1 Set high to stop merge engine logic 100 from
    running
    AssertException 1 Set high to assert exception
    Reset
    1 Set high to reset merge engine
    ResetFrame
    1 Set high to reset current frame
    Snapshot
    1 Set high to snapshot all internal states
    NoPropagate
    1 Set high to prevent propagation of control
    messages
  • Registers [0088] 144 also include a snapshot register, an M-byte register, which holds an M-byte result of a last snapshot request. The snapshot register is readable when the merge engine is stopped. Registers 144 also include an ASIC address register and an ASIC data register used for accessing the state of ASIC 102. The ASIC address register is an eight-byte register that indicates a target of the data held in the ASIC data register.
  • Two other registers in registers [0089] 114 are host readable and writeable, but only when the merge engine is stopped. Those two registers are an address register, the fields of which are shown in Table 3, and a control/status register, the fields of which are shown in Table 4.
    TABLE 3
    ADDRESS REGISTER
    Field Description
    HostInRgb Base address for reading frame buffer pixels
    HostInAlpha Base address for reading frame buffer alpha
    HostInZ Base address for reading Z buffer
    HostOutRgb Base address for writing frame buffer pixels
    HostOutAlpha Base address for writing frame buffer alpha
    HostOutZ Base address for writing Z buffer
  • [0090]
    TABLE 4
    CONTROL/STATUS REGISTER
    Field # of bits Description
    Xres
    16 Horizontal image resolution, in pixels.
    Yres 16 Vertical resolution, in pixels
    X
    16 Current value of X
    Y
    16 Current value of Y
    FUCoreMode
    32 Inclusive OR of operating modes (Z compare,
    blend, reserved*)
  • An image generation operation using the image generator of FIG. 1 with the merge engine of FIG. 2 will now be described with reference to FIG. 3. There, a four stage image generator is shown (i.e., N=4). For simplicity, the [0091] merge engines 24 are shown with direct connections, but it should be understood that these connections might well be via pixel bus 28. Each stage, or thread, of the parallel image generation process comprises a rendering engine (RE) 22 and a merge engine (ME) 24. Each stage might be implemented on a different host running different threads of a multithreaded program, as explained below in connection with FIG. 4.
  • In this example, each stage processes one shape and those shapes are a circle, two rectangles and a triangle. Image controller [0092] 20 (see FIG. 1) passes an object description of the circle to rendering engine 22(1) along with a command to initiate rendering of the circle. Typically, the object description of a circle includes a location for the center of the circle and a radius in the screen coordinate space, along with a description of the surface of the circle. The description of the circle's surface could be as simple as a solid color, but more often it is a complex description, possibly including a texture or a description of the reflectance/transmission properties assigned to the object and a description of the light sources or shadows impinging on the object.
  • [0093] Image controller 20 also passes the three other objects to their respective stages for parallel processing. As rendering engine 22(1) determines blocks of pixel color values, it outputs those blocks as a pixel stream to merge engine 24(1) and eventually outputs all pixel values comprising part image 170(1). Rendering engines 22(2)-(4) do the same, outputting part images 170(2)-(4) in parallel as they are generated.
  • Along with the pixel values, [0094] rendering engines 22 preferably also output Z values for the pixels, where a Z value for a pixel indicates the depth of the portion of object represented by that pixel. Merge engines 24 use the Z values to determine when one object obscures another. For the purposes of illustration, each of the objects are placed at constant depths, with objects OBJ1-OBJ4 being at depths of 1, 3, 2 and 4, respectively. Here, a lower Z represents a depth further from the view point. This is reflected in the fact that in the final image, OBJ1 (the circle at z=1) is obscured by OBJ2 (the rectangle at x=3) and OBJ3 (the rectangle at x=2).
  • In one embodiment, each stage is supported by one workstation and various threads of a multiprocessing program are parsed out to the workstations. FIG. 4 shows details of one such system. There, one workstation controls the overall image generation process by running a program thread (TH[0095] 0) that communicates with threads in other workstations 180 over a network interconnecting the workstations. API0 distributes rendering commands from TH0 among the set of threads TH1 through TH N 190 running on workstations 1 through N. Each thread THi relays the rendering commands, with minimal overhead, to an API i 192 on workstation i, which can be an unmodified OpenGL API that drives a single graphics accelerator card GA i 194. In this configuration, off-the-shelf workstations and graphics accelerator cards can be used for image generation, which greatly reduces the cost of the system. The marginal costs of the image generation process might be reduced even further if the workstations and graphics accelerators are used for other purposes. All that would normally be needed to be added is suitable software for control of the various elements of the system and N merge engines with their connections to a network such as a ServerNet® network bus. Since an existing ASIC can be used to interface the merge engines to the network bus, as explained above, design costs are minimized.
  • Referring again to FIG. 4, a user or user application interacts with process thread TH[0096] 0. Thread TH0 accesses the set of N graphics accelerator cards (GAi) 194 housed in some N workstations (or fewer than N if multiple graphics accelerator cards can be installed in one or more workstations) via application program interface API0 on workstation 0 that is running TH0. API0 distributes the rendering information to the graphics accelerators by communicating with a set of N threads (THi). Each thread invokes a local API to drive a graphics accelerator to produce an image in a frame buffer and this image is provided to the thread's corresponding merge engine. This image is passed over a private bus to a merge engine (e.g., merge engine 24(i)) from it is fed into a ServerNet®-based pathway of merge engines. The final merge engine, 24(N) deposits the fully processed image back into the final frame buffer, or a dedicated frame buffer, from where the image can be sent to a display.
  • Using the workstation arrangement shown in FIG. 4, there are several data paths that might be bottlenecks under certain conditions. The bandwidth of the various data paths should be considered in the design of an image generator so that a bottleneck does not eliminate the speed benefits of other enhancements. [0097]
  • One data path of concern is the network path between API[0098] 0 and the various threads. A typical graphics accelerator today has a peak throughput of a few million triangles/second, but assuming a realistic utilization of 25%, or 250,000 triangles, the bandwidth from API0 to each thread 190 would be between 2.5 and 5 MBS, assuming that texture data has been preloaded and is locally available to each accelerator card 194. If multiple accelerators are housed in a single multiprocessor workstation, this bandwidth can be easily supported through the memory system. When API0 is in a separate workstation from the threads, local network links can easily carry the data traffic, over a ServerNet® network bus or some other network, so that data path is not a problem.
  • If the links between [0099] workstation 0 and workstations 1 through N cannot handle the required bandwidth, the application might be designed with explicit parallelism in mind, either as a SPMD (Single Program, Multiple Data) program or MIMD (Multiple Instruction, Multiple Data) program. Using that approach, thread TH0 and API0 cooperate to send high level instructions to threads 190 that are application threads. The model data is distributed in such a way that it is available locally to threads 190 and does not have to be transferred between API0 and threads 190. Using this approach, thread TH0 sends instructions to the other threads 190 to manipulate the model or viewing position and threads 190 perform local updates to the model data and initiate the rendering commands that drive their APIs 192. That way, the image generation is not limited by the inter-workstation bandwidth, since a model can be, for example, rotated in real-time simply by broadcasting a change of viewing position, which can be defined by 24 bytes. The bandwidth needed between API0 and threads 190 is only these 24 bytes, since the model itself is located on the workstations that contain the graphics accelerators.
  • The above-described architecture is easily realized by a cluster computer configuration in which all of the workstations are connected by a ServerNet® network bus (which can be the same ServerNet® network bus as the bus carrying merge engine-to-merge engine traffic, or a separate bus). [0100]
  • In such a configuration, the connections among the merge engines could be realized by the ServerNet® links and switches. Although such a configuration would be more expensive than a pipelined configuration it would be a natural way to perform rendering on a cluster computer that was already ServerNet®-connected. It would also be an appropriate way to build a fault-tolerant rendering system, and this could be important for real-time applications that require high reliability, such as flight simulators and location based entertainment. For visualizing scientific applications on cluster computers the most desirable configuration would combine a pipelined architecture, using dedicated ServerNet® links, with a general purpose ServerNet® network used for all other communication. [0101]
  • In the above descriptions of the merge pathway (i.e., the order of traffic flow between merge engines), the pathway is from merge engine [0102] 24(1) to merge engine 24(2), and so on, to merge engine 24(N) and then to merge engine 24(1). In another embodiment, illustrated in FIG. 5, the order of the merge engines in the pathway is not fixed, but is controlled by a switch network 200. In some cases, switch network 200 might be a physical message router, while in other cases, switch network 200 is implemented by dynamically modifying the merge engines' addresses on pixel bus 28 or the destinations of their message.
  • One use of a dynamically ordered merge pathway is to allow for noncumulative operations, as might be needed for handling fog and transparency in an image. In some implementations, it may be appropriate to combine pipelined and switched configurations. The pipelined configurations are scalable and inexpensive, while the fault tolerant configuration is appropriate for high reliability applications including flight simulation and location based entertainment. The fault tolerant configuration provides double the link bandwidth of the pipelined configuration because it allows ServerNet® channels to communicate in two directions simultaneously. [0103]
  • In switched configurations, the merge engines (specifically, the x and y ports of ASIC [0104] 102) are connected to a ServerNet® switch that is a 300 nanosecond latency crossbar with 15 gigabit bisection bandwidth (in current implementations). Data flows bi-directionally on both the x and y links, with the upstream and downstream traffic split across the two links. In switched configurations that use bi-directional pathways, the arbiter is responsible for merging the results of an incoming stream that has been split across two links.
  • The above description illustrates several examples and variations of a parallel image generator using pipelined merge engines. Several other considerations should be taken into account to ensure that the best performance is obtained. For example, in building a system using off-the-shelf graphics accelerators, the overall system is constrained somewhat by the functions of those graphics accelerators. For example, if fog and alpha blending are required of the overall system, the system should be designed with graphics accelerators that handle those functions, since a merge engine might not be able to handle fog and alpha blending if the graphics accelerator does not provide enough information, such as alpha data. [0105]
  • As with any parallel computation, the effective speedup will be limited by the achievable load balance. A virtue of the image composition approach described herein to achieving scalability, in contrast to other architectures, is that there is no subdivision of the image at any point in the architecture. This makes it possible to distribute the computational work using a randomization strategy. Of course, if there are bandwidth or other constraints, dividing the image up and having more than one independent image generator work on each subimage might be desirable. From the central limit theorem, an expected value of load balance under such strategies can be predicted if the variance of the workload distribution is known. It is possible to predict effective load balance using off-line simulation, and this would be valuable in order to verify the operation of the system as well as to predict performance. [0106]
  • Yet another consideration is frame rate/resolution. Image resolutions range from a low of 640×480 pixels, used in video games and current television standards, through 2000×1000 for HDTV imagery, and higher for flight simulators and high-end film animation. Table 5 identifies some target frame rates and resolutions along with corresponding bandwidth requirements. The data requirements vary from 5 to 8 bytes per pixel depending on the resolution of the Z buffer (2, 3 or 4 bytes) and whether an alpha channel is included. [0107]
    TABLE 5
    DATA
    PER TARGET
    RES- FRAME FRAMES/ BANDWIDTH
    APPLICATION OLUTION (MB) SEC (MBS)
    Games 640 × 480 1.5-2.5 30 45-75
    CAD 1280 × 1024  6.5-10.5 10  65-105
    HDTV 2000 × 1000 10-16 30 300-480
    Flight simulators 1700 × 1350 11.5-18.5  30+ 345-555
  • Where frame rates are limited by available bandwidth, the bandwidth can be increased by using multiple rendering systems with duplicate model data, with each system rendering a portion of each frame. The results are then combined to make a full frame. Alternately (but perhaps more problematically) each system could render alternating frames in a round-robin sequence. [0108]
  • Bandwidth requirements decrease for decreasing target frame rates. This may be a reasonable compromise for some applications, such as high quality animation, where a 20 frame-per-second solution is useful (although less valuable than a 30 frame-per-second solution). This compromise should be acceptable for non-interactive applications. [0109]
  • In addition to bandwidth concerns, latency should also be considered. The rendering process incurs latency in three stages: first, while application threads prepare data for the graphics accelerators; second, while graphics accelerators generate pixel and Z data for the frame buffers; and third, while the frame buffer data flows through the pathway of merge engines to produce a final image. If the combined latency from these stages is too high (roughly, above 100 ms) then interactive users will experience motion sickness. Since the threads will introduce some latency, and the graphics accelerators may typically introduce additional latency, the merge pathway should nominally operate within a 30 ms time window. The total pathway latency, LT, for an N stage pipeline is: [0110]
  • L T =L×(N+P−1)
  • where L is the latency of each stage and P is the number of packets or pixels being processed. Although the merge pathway will work with packets that are multiples of 64 bytes each, an equivalent expression can be stated in terms of pixels. The above expression can be used to evaluate the maximum latency, L, per pixel that can be tolerated while processing images with varying numbers of pixels. The total pathway latency, L[0111] T, is approximately L×P since (N−1)<<P. Table 6 shows this maximum latency per pixel and the target frequency with which pixels would have to be processed, in order to achieve 30 ms latency in the merge pathway for given resolutions.
    TABLE 6
    Resolution Latency/pixel Frequency
    640 × 480 97.6 ns 11 MHz
    1280 × 1024 22.9 ns 44 MHz
    2000 × 1000 15.0 ns 67 MHz
    1700 × 1350 13.1 ns 77 MHz
  • Those frequency targets can be achieved in three ways: by adjusting the clock rate of [0112] merge engine logic 100 subject to available bandwidth on pixel bus 28; by parallelizing the data pathways in merge engine logic 100 (subject again to available pixel bus 28 bandwidth); or by deploying multiple merge pathways in parallel, subject to available bandwidth through the PCI bus and host subsystems.
  • In the first generation of the Colorado ASIC, 38 MBS of unidirectional raw data bandwidth is available. In the next generation, 230 MBS will be available bi-directionally, rising to over 1 GBS in subsequent generations, which should accommodate most applications. Even with the first generation, if more bandwidth is needed, one approach is to deploy multiple merge pathways in parallel. It is probably feasible to use two such pathways in a dual peer PCI host like the Compaq workstation [0113] 6000 or Digital Alpha workstations.
  • Another approach is to use data compression between the merge engines, as described in an above-referenced co-filed patent application, to increase the effective bandwidth of [0114] pixel bus 28.
  • FIGS. 6-8 illustrate a specific implementation of a graphics processor according to the present invention. FIG. 6 is a high-level block diagram of the components that make up one [0115] node processor 300. In this implementation, node processor 300 is hosted by a workstation (not shown) that is connected to other workstations via a ServerNet® network, with logical channels coupling processor node 300 to its two nearest neighbors, a previous workstation and a next workstation. As explained above, a processor node can be a rendering node, a display node or both. Also, as explained above, the order of the nodes can be dynamically altered by altering the logical connections between workstations.
  • As shown in FIG. 6, [0116] node processor 300 includes a CPU 302, coupled via a host bridge 303 to a host memory 304, a 3D accelerator card 306 and a compositing card 308. The coupling is provided by a host PCI bus 309. 3D accelerator 306 is shown comprising a 3D rendering chip 310 and a graphics memory 312. Compositing card 308 is shown comprising a merge engine chip 320 and a network interface chip 322 (a Colorado ASIC for interfacing to a ServerNet® network).
  • The standard data path within [0117] node processor 300 is shown by the numbered arrows. Using its local data or data provided externally, CPU 302 prepares 3D data and stores it in host memory 304 (arrow 1). That data is sent, as OpenGL commands, over host PCI bus 309 to 3D accelerator card 306 (arrow 2). 3D rendering chip 310 then executes the OpenGL commands and renders 3D objects to graphics memory 312 (arrow 3).
  • In a typical graphics processor, at this point the color data are read out of [0118] memory 312 and displayed. However, here the data is sent to the merge engine (arrow 4). The preferred way for merge engine 320 to read data from graphics memory 312 is through a private interface as shown in FIG. 6. This method is preferred because it alleviates the host PCI bus from supporting the bandwidth required for this data transfer. Another way for merge engine 320 to read data from graphics memory 312 is to program merge engine 320 to perform direct memory access (DMA) using the PCI interface, rather than having 3D accelerator 306 DMA the data or doing a transfer via a main memory. While this is a good approach if such capability is available, it might not always be available. For example, if and AGP 3D accelerator that implements Intel's AGPSets, PCI-to-AGP DMA reads would not be supported.
  • In experiments, the best performance of a 3D accelerator card that uses the Evans & Sutherland RealImage 2100 chip set is obtained with bursts of four 32-bit words and a number of wait states between transactions that depends on the direction of the data transfer. As explained above, the typical data set includes color (RGB), transparency (a) and depth (z). Preferably, this data set is received in a scan-line fashion simultaneously from the [0119] graphics memory 312 and from the upstream node (arrow 5). The merge function actually applied to the local data and the upstream data is user-configurable, as explained above. Merge engine 320 applies the merge function in use to the local data and the upstream data and then sends the resulting merged data set to the downstream node (arrow 6).
  • [0120] Processor node 300, in this specific embodiment, is a Compaq workstation, such as an IA32, or an Alpha processor-based workstation running Microsoft's Windows NT operating system, 3D accelerator card 306 is a Compaq PowerStorm™ 300 3D accelerator, and compositing card 308 comprises a Compaq PCI Pamette card (marketed as Compaq's “PCI Development Platform”), version 1, revision 2, using 4 user-programmable FPGA's (part number XC4044XL-1, marketed by Xilinx). Compositing card 308 also includes a custom Compaq ServerNet® PMC daughter card for the PCI to ServerNet® network interface.
  • In a network of multiple processor nodes, the task of rendering is spread over all the processor nodes. In a “sort-last-full” approach, each processor node operates on the entire image, but only with a subset of the objects. The merge engines then merge these individual images together. In a “sort-middle” approach, each processor node processes all the objects, but only for a part of the screen attributed to that processor node. In that case, the merge engines are used to tile together the results from the individual processor nodes. The latter approach is useful for parallel ray tracing. [0121]
  • The logical topology for the network is the chain, wherein each node has one input data stream and one output data stream, but other topologies can be used, such as a loop, or a tree. A loop, or ring, is just a chain with the two chain ends closed to each other. A loop is useful where the data has to be recirculated in the rendering system. While a tree provides lower latency, it requires more network bandwidth, which is often a scarce resource. However the nodes are connected, it can be useful to connect all the network inputs and outputs to a switch so that the order of the nodes can be arbitrarily reordered. Dynamic reordering is useful where merging operations require a specific order and that order changes, such as when the viewpoint changes. [0122]
  • FIG. 7 shows the components of [0123] merge engine 320 in more detail, including a separation of various components according to which of the four FPGA areas 330(1)-(4) is used for the components.
  • A graphics system using [0124] processor nodes 300 can pipeline processing very aggressively. With double buffering for the color and depth buffers, pipelining as shown in FIG. 8 can be obtained. On each node, rendering can start as soon as both the previous rendering and the previous merging are finished. Merging can begin as soon as the previous rendering and the previous merging are finished, but may stall if upstream data is not received or there is no space downstream to store the results (shown in FIG. 8 by cross-hatching). Because of the node-to-node flow control, no broadcast is needed to stop and start the data flow. This way, even on a system with a high node count (such as 128 nodes or more), stalls stay local and the data flow resumes with a latency of one node.
  • The above description is illustrative and not restrictive. Many variations of the invention will become apparent to those of skill in the art upon review of this disclosure. The scope of the invention should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the appended claims along with their full scope of equivalents. [0125]

Claims (38)

What is claimed:
1. A system for generating images comprising:
a pixel bus;
a host bus;
a plurality of merge engines, each merge engine comprising:
a pixel bus interface circuit configured to send and receive pixel data and control messages over the pixel bus between merge engines;
a part image bus configured to communicate with a render engine associated the merge engine; and
a host interface configured to send and receive pixel data and control messages over the host bus between the merge engine and a host system.
2. The system of claim 1 wherein the control messages over the host bus between the merge engine and a host system comprise synchronization messages.
3. The system of claim 1 wherein the control messages over the host bus between the merge engine and a host system comprise initialization messages.
4. The system of claim 1 wherein the merge engine further comprises a plurality of control registers.
5. The system of claim 4 wherein the control messages over the host bus between the merge engine and a host system comprise control data to be written to the control registers.
6. The system of claim 4 wherein the control messages over the pixel bus between merge engines comprise control data to be written to the control registers.
7. The system of claim 4 wherein the control registers comprise a first plurality of control registers and a second identical plurality of control registers.
8. The system of claim 7 wherein only the first plurality of registers can be modified by messages on the pixel bus.
9. A graphics merge engine comprising:
a pixel bus interface circuit configured to send and receive pixel data and control messages over a pixel bus between merge engines;
a part image bus configured to communicate with a render engine associated the merge engine;
a host interface configured to send and receive pixel data and control messages over a host bus between the merge engine and a host system; and
a pixel data compare circuit in communication with the pixel bus interface, part image bus, and host interface.
10. The graphics merge engine of claim 9 further comprising a plurality of control registers.
11. The graphics merge engine of claim 10 wherein the plurality of control registers comprises a first plurality and a second identical plurality of control registers.
12. The graphics merge engine of claim 11 wherein only the first plurality of control registers can be modified by messages on the pixel bus.
13. A graphics processing system comprising:
a host processor;
a rendering engine;
a graphics memory in communication with the rendering engine;
a merge processor in communication with the graphics memory, the merge processor comprising:
a pixel bus interface circuit configured to send and receive pixel data and control messages over the pixel bus between other merge processors and workstations;
a host interface configured to send and receive pixel data and control messages over the host bus between the merge processor and the host processor; and
a pixel merge circuit.
14. The system of claim 13 wherein the control messages over the host bus between the merge processor and the host processor comprise synchronization messages.
15. The system of claim 13 wherein the control messages over the host bus between the merge processor and the host processor comprise initialization messages.
16. The system of claim 13 wherein the merge processor further comprises a plurality of control registers.
17. The system of claim 16 wherein the control messages over the host bus between the merge processor and the host processor comprise control data to be written to the control registers.
18. The system of claim 16 wherein the control messages over the pixel bus between merge processors comprise control data to be written to the control registers.
19. The system of claim 16 wherein the control registers comprise a first plurality of control registers and a second identical plurality of control registers.
20. The system of claim 19 wherein only the first plurality of registers can be modified by the messages on the pixel bus.
21. A graphics processing system comprising:
a pixel bus for communicating data comprising pixel information;
a plurality of merge processors for merging pixel information received from at least one neighbor processor;
a plurality of rendering processors in communication with at least one merge processor and for rendering pixel information; and
a switch network for assigning and reassigning which processors are input neighbors and which processors are output neighbors.
22. The system of claim 21 wherein at least one merge processor comprises an identifying address on the pixel bus and wherein the identifying address is modifiable by the switch network.
23. The system of claim 21 wherein each merge processor comprises an identifying address on the pixel bus and wherein the identifying address is modifiable by the switch network to define which processors are input neighbors and which processors are output neighbors.
24. A graphics processing system comprising:
a pixel bus for communicating data comprising pixel information;
a plurality of merge processors for merging pixel information received from at least one neighbor processor, the plurality of merge processors forming a first state of input neighbor and output neighbor associations for a first image and a second state of input neighbor and output neighbor associations for a second image;
a plurality of rendering processors in communication with at least one merge processor and for rendering pixel information; and
a switch network for changing between the first state of input neighbor and output neighbor associations for the first image and the second state of input neighbor and output neighbor associations for the second image.
25. The system of claim 24 wherein the first state comprises a pipeline configuration between a merge processor, its input neighbor processor, and its output neighbor processor.
26. The system of claim 24 wherein the first state of input neighbor and output neighbor associations comprises a first processor and a second processor and wherein the first processor is an input processor to the second processor and wherein the second state of input neighbor and output neighbor associations comprises the first processor being an output neighbor to the second processor.
27. The system of claim 24 wherein the first state of input neighbor and output neighbor associations comprises a first processor and a second processor and wherein the first processor is an input neighbor to the second processor and wherein the second state of input neighbor and output neighbor associations comprises the first processor not being an input or output neighbor to the second processor.
28. The system of claim 24 wherein at least one merge processor comprises an identifying address on the pixel bus.
29. The system of claim 24 wherein each merge processor comprises an identifying address on the pixel bus and wherein the identifying address is modifiable by the switch network to define which processors are input neighbors and which processors are output neighbors.
30. The system of claim 24 wherein each merge processor comprises at least one configurable neighbor processor input that can receive pixel information from any one or more definable input neighbor processors and at least one dynamically configurable neighbor processor output that can output pixel information to any one or more definable output neighbor processors.
31. A system for generating images comprising:
a plurality of merge engines comprising at least one merge engine having a plurality of configuration registers that identify a plurality of operating modes of the merge engine;
a host in communication with at least the at least one merge engine and arranged to write data to the configuration registers to configure the merge engine to operate in at least a first mode of image analysis.
32. The system of claim 31 wherein the host further comprises logic for writing data to the configuration registers to configure the merge engine to operate in at least a second mode of image analysis.
33. The system of claim 31 wherein the first mode of image analysis comprises a pixel depth comparison mode of operation.
34. The system of claim 31 wherein the first mode of image analysis comprises a pixel blending mode of operation.
35. The system of claim 31 wherein the first mode of image analysis comprises a pixel data pass-through mode of operation.
36. A network comprising:
a plurality of workstations in communication with each other, at least one work station comprising:
a memory;
an image rendering engine;
an image merge engine in communication with the rendering engine and comprising:
an input neighbor merge engine data bus;
an output neighbor merge engine data bus; and
a plurality of registers that define a first state of input neighbor merge engine and output neighbor merge engine associations and wherein the plurality of registers are modifiable to define a second state of input neighbor merge engine and output neighbor merge engine associations different from the first.
37. The network of claim 36 wherein an image merge engine associated with a first workstation comprises an input neighbor merge engine to an image merge engine associated with a second workstation.
38. The network of claim 36 wherein an image merge engine associated with a first workstation comprises an output neighbor merge engine to an image merge engine associated with a second workstation.
US10/864,609 1999-03-08 2004-06-09 Parallel pipelined merge engines Abandoned US20040223003A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10/864,609 US20040223003A1 (en) 1999-03-08 2004-06-09 Parallel pipelined merge engines

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US09/264,347 US6753878B1 (en) 1999-03-08 1999-03-08 Parallel pipelined merge engines
US10/864,609 US20040223003A1 (en) 1999-03-08 2004-06-09 Parallel pipelined merge engines

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US09/264,347 Continuation US6753878B1 (en) 1999-03-08 1999-03-08 Parallel pipelined merge engines

Publications (1)

Publication Number Publication Date
US20040223003A1 true US20040223003A1 (en) 2004-11-11

Family

ID=32467692

Family Applications (2)

Application Number Title Priority Date Filing Date
US09/264,347 Expired - Lifetime US6753878B1 (en) 1999-03-08 1999-03-08 Parallel pipelined merge engines
US10/864,609 Abandoned US20040223003A1 (en) 1999-03-08 2004-06-09 Parallel pipelined merge engines

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US09/264,347 Expired - Lifetime US6753878B1 (en) 1999-03-08 1999-03-08 Parallel pipelined merge engines

Country Status (1)

Country Link
US (2) US6753878B1 (en)

Cited By (64)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030210351A1 (en) * 2002-05-13 2003-11-13 Kozo Murakami Superimposing apparatus and method for broadcasting three-dimensional computer graphics image
US20040054657A1 (en) * 2002-09-13 2004-03-18 Konica Minolta Holdings, Inc. Medical information management system
US20050015718A1 (en) * 2003-07-16 2005-01-20 Sambhus Mihir Y. Method and system for client aware content aggregation and rendering in a portal server
US20050012749A1 (en) * 2003-07-15 2005-01-20 Nelson Gonzalez Multiple parallel processor computer graphics system
US20050088445A1 (en) * 2003-10-22 2005-04-28 Alienware Labs Corporation Motherboard for supporting multiple graphics cards
US20050231514A1 (en) * 2004-04-16 2005-10-20 John Harper System for optimizing graphics operations
US20050231521A1 (en) * 2004-04-16 2005-10-20 John Harper System for reducing the number of programs necessary to render an image
US20050235287A1 (en) * 2004-04-16 2005-10-20 John Harper System for emulating graphics operations
US20050285965A1 (en) * 2004-06-24 2005-12-29 Apple Computer, Inc. User-interface design
US20050285867A1 (en) * 2004-06-25 2005-12-29 Apple Computer, Inc. Partial display updates in a windowing system using a programmable graphics processing unit
US20050285866A1 (en) * 2004-06-25 2005-12-29 Apple Computer, Inc. Display-wide visual effects for a windowing system using a programmable graphics processing unit
US20060232590A1 (en) * 2004-01-28 2006-10-19 Reuven Bakalash Graphics processing and display system employing multiple graphics cores on a silicon chip of monolithic construction
US20060285735A1 (en) * 2005-06-10 2006-12-21 Ulrich Hagen Method and device for controlling image data acquisition and image reconstruction
US20060284878A1 (en) * 2004-06-24 2006-12-21 Apple Computer, Inc. Resolution Independent User Interface Design
US20070035559A1 (en) * 2005-08-12 2007-02-15 Microsoft Corporation Compositing external images into a multimedia rendering pipeline
US20070076007A1 (en) * 2005-09-21 2007-04-05 Quanta Computer Inc. Display controller capable of reducing cache memory and the frame adjusting method thereof
US20070182747A1 (en) * 2004-04-16 2007-08-09 John Harper High-level program interface for graphics operations
US20070242819A1 (en) * 2004-04-30 2007-10-18 Siemens Aktiengesellschaft Method and Unit Used to Determine Useable Services
US20070247468A1 (en) * 2004-04-16 2007-10-25 Mark Zimmer System and method for processing graphics operations with graphics processing unit
US20070279411A1 (en) * 2003-11-19 2007-12-06 Reuven Bakalash Method and System for Multiple 3-D Graphic Pipeline Over a Pc Bus
US20080117217A1 (en) * 2003-11-19 2008-05-22 Reuven Bakalash Multi-mode parallel graphics rendering system employing real-time automatic scene profiling and mode control
US20080158236A1 (en) * 2006-12-31 2008-07-03 Reuven Bakalash Parallel graphics system employing multiple graphics pipelines wtih multiple graphics processing units (GPUs) and supporting the object division mode of parallel graphics rendering using pixel processing resources provided therewithin
US20080165186A1 (en) * 2007-01-05 2008-07-10 Landmark Graphics Corporation, A Halliburton Company Systems and methods for visualizing multiple volumetric data sets in real time
US20080165190A1 (en) * 2007-01-09 2008-07-10 Samsung Electronics Co., Ltd. Apparatus and method of displaying overlaid image
US20080211816A1 (en) * 2003-07-15 2008-09-04 Alienware Labs. Corp. Multiple parallel processor computer graphics system
US20080250227A1 (en) * 2007-04-04 2008-10-09 Linderman Michael D General Purpose Multiprocessor Programming Apparatus And Method
US20090106530A1 (en) * 2007-10-17 2009-04-23 Christian Lauterbach System, method, and computer program product for generating a ray tracing data structure utilizing a parallel processor architecture
US7542042B1 (en) * 2004-11-10 2009-06-02 Nvidia Corporation Subpicture overlay using fragment shader
US20090303233A1 (en) * 2008-06-06 2009-12-10 Landmark Graphics Corporation, A Halliburton Company Systems and Methods for Imaging a Three-Dimensional Volume of Geometrically Irregular Grid Data Representing a Grid Volume
WO2009154484A2 (en) * 2008-06-20 2009-12-23 Business Intelligence Solutions Safe B.V. Methods, apparatus and systems for data visualization and related applications
US20100033502A1 (en) * 2006-10-13 2010-02-11 Freescale Semiconductor, Inc. Image processing apparatus for superimposing windows displaying video data having different frame rates
US7681112B1 (en) 2003-05-30 2010-03-16 Adobe Systems Incorporated Embedded reuse meta information
US20100085441A1 (en) * 2008-10-08 2010-04-08 Fuji Xerox Co., Ltd. Image processing apparatus, image processing system and computer readable medium
US7707514B2 (en) 2005-11-18 2010-04-27 Apple Inc. Management of user interface elements in a display environment
US20100149195A1 (en) * 2008-12-15 2010-06-17 Canon Kabushiki Kaisha Load balancing in multiple processor rendering systems
US7743336B2 (en) 2005-10-27 2010-06-22 Apple Inc. Widget security
US7752556B2 (en) 2005-10-27 2010-07-06 Apple Inc. Workflow widgets
US7761800B2 (en) 2004-06-25 2010-07-20 Apple Inc. Unified interest layer for user interface
US7777748B2 (en) 2003-11-19 2010-08-17 Lucid Information Technology, Ltd. PC-level computing system with a multi-mode parallel graphics rendering subsystem employing an automatic mode controller, responsive to performance data collected during the run-time of graphics applications
US7793232B2 (en) 2004-06-25 2010-09-07 Apple Inc. Unified interest layer for user interface
US7954064B2 (en) 2005-10-27 2011-05-31 Apple Inc. Multiple dashboards
US7961194B2 (en) 2003-11-19 2011-06-14 Lucid Information Technology, Ltd. Method of controlling in real time the switching of modes of parallel operation of a multi-mode parallel graphics processing subsystem embodied within a host computing system
US8140975B2 (en) 2005-01-07 2012-03-20 Apple Inc. Slide show navigation
US20120069023A1 (en) * 2009-05-28 2012-03-22 Siliconarts, Inc. Ray tracing core and ray tracing chip having the same
US8156467B2 (en) 2007-08-27 2012-04-10 Adobe Systems Incorporated Reusing components in a running application
US8176466B2 (en) 2007-10-01 2012-05-08 Adobe Systems Incorporated System and method for generating an application fragment
US8239749B2 (en) 2004-06-25 2012-08-07 Apple Inc. Procedurally expressing graphic objects for web pages
US8284207B2 (en) 2003-11-19 2012-10-09 Lucid Information Technology, Ltd. Method of generating digital images of objects in 3D scenes while eliminating object overdrawing within the multiple graphics processing pipeline (GPPLS) of a parallel graphics processing system generating partial color-based complementary-type images along the viewing direction using black pixel rendering and subsequent recompositing operations
US8302020B2 (en) 2004-06-25 2012-10-30 Apple Inc. Widget authoring and editing environment
US20130100498A1 (en) * 2011-10-25 2013-04-25 Canon Kabushiki Kaisha Image formation processing apparatus and image processing method
US8453065B2 (en) 2004-06-25 2013-05-28 Apple Inc. Preview and installation of user interface elements in a display environment
US8543824B2 (en) 2005-10-27 2013-09-24 Apple Inc. Safe distribution and use of content
US8543931B2 (en) 2005-06-07 2013-09-24 Apple Inc. Preview including theme based installation of user interface elements in a display environment
US8566732B2 (en) 2004-06-25 2013-10-22 Apple Inc. Synchronization of widgets and dashboards
US8656293B1 (en) 2008-07-29 2014-02-18 Adobe Systems Incorporated Configuring mobile devices
US8667415B2 (en) 2007-08-06 2014-03-04 Apple Inc. Web widgets
US20140168231A1 (en) * 2012-12-18 2014-06-19 Nvidia Corporation Triggering performance event capture via pipelined state bundles
US8869027B2 (en) 2006-08-04 2014-10-21 Apple Inc. Management and generation of dashboards
US8954871B2 (en) 2007-07-18 2015-02-10 Apple Inc. User-centric widgets and dashboards
US9104294B2 (en) 2005-10-27 2015-08-11 Apple Inc. Linked widgets
US20160148339A1 (en) * 2014-11-20 2016-05-26 Intel Corporation Apparatus and method for efficient frame-to-frame coherency exploitation for sort-last architectures
US20160210159A1 (en) * 2015-01-19 2016-07-21 Microsoft Microsoft Technology Licensing, LLC User Mode Driver Extension and Preprocessing
US9619304B2 (en) 2008-02-05 2017-04-11 Adobe Systems Incorporated Automatic connections between application components
US9817776B2 (en) 2015-01-19 2017-11-14 Microsoft Technology Licensing, Llc Memory descriptor list caching and pipeline processing

Families Citing this family (67)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000153647A (en) * 1998-11-18 2000-06-06 Canon Inc Image-processing apparatus and its method
US6924807B2 (en) * 2000-03-23 2005-08-02 Sony Computer Entertainment Inc. Image processing apparatus and method
GB0007974D0 (en) * 2000-04-01 2000-05-17 Discreet Logic Inc Processing image data
JP3688618B2 (en) * 2000-10-10 2005-08-31 株式会社ソニー・コンピュータエンタテインメント Data processing system, data processing method, computer program, and recording medium
US6526491B2 (en) * 2001-03-22 2003-02-25 Sony Corporation Entertainment Inc. Memory protection system and method for computer architecture for broadband networks
US7239324B2 (en) * 2001-03-23 2007-07-03 Microsoft Corporation Methods and systems for merging graphics for display on a computing device
US7038690B2 (en) * 2001-03-23 2006-05-02 Microsoft Corporation Methods and systems for displaying animated graphics on a computing device
US6919900B2 (en) * 2001-03-23 2005-07-19 Microsoft Corporation Methods and systems for preparing graphics for display on a computing device
JP3970102B2 (en) * 2001-06-28 2007-09-05 キヤノン株式会社 Image processing apparatus, image processing method, and program
US7088467B1 (en) * 2001-07-06 2006-08-08 Hewlett-Packard Development Company, L.P. Digital video imaging with high-resolution still imaging capability
US20030061269A1 (en) * 2001-09-17 2003-03-27 Flow Engines, Inc. Data flow engine
US6714203B1 (en) * 2002-03-19 2004-03-30 Aechelon Technology, Inc. Data aware clustered architecture for an image generator
WO2003083680A1 (en) * 2002-03-22 2003-10-09 Deering Michael F Scalable high performance 3d graphics
US7451457B2 (en) * 2002-04-15 2008-11-11 Microsoft Corporation Facilitating interaction between video renderers and graphics device drivers
US7219352B2 (en) * 2002-04-15 2007-05-15 Microsoft Corporation Methods and apparatuses for facilitating processing of interlaced video images for progressive video displays
EP1359773B1 (en) * 2002-04-15 2016-08-24 Microsoft Technology Licensing, LLC Facilitating interaction between video renderers and graphics device drivers
US9377987B2 (en) * 2002-10-22 2016-06-28 Broadcom Corporation Hardware assisted format change mechanism in a display controller
US7061495B1 (en) * 2002-11-18 2006-06-13 Ati Technologies, Inc. Method and apparatus for rasterizer interpolation
US7796133B1 (en) 2002-11-18 2010-09-14 Ati Technologies Ulc Unified shader
US8933945B2 (en) * 2002-11-27 2015-01-13 Ati Technologies Ulc Dividing work among multiple graphics pipelines using a super-tiling technique
US7633506B1 (en) 2002-11-27 2009-12-15 Ati Technologies Ulc Parallel pipeline graphics system
US8749561B1 (en) * 2003-03-14 2014-06-10 Nvidia Corporation Method and system for coordinated data execution using a primary graphics processor and a secondary graphics processor
US20040179007A1 (en) * 2003-03-14 2004-09-16 Bower K. Scott Method, node, and network for transmitting viewable and non-viewable data in a compositing system
US7034837B2 (en) * 2003-05-05 2006-04-25 Silicon Graphics, Inc. Method, system, and computer program product for determining a structure of a graphics compositor tree
US9946779B2 (en) * 2003-05-28 2018-04-17 Oracle International Corporation Pipleline merge operations using source data and multiple destination data structures
US7015913B1 (en) * 2003-06-27 2006-03-21 Nvidia Corporation Method and apparatus for multithreaded processing of data in a programmable graphics processor
US7158668B2 (en) 2003-08-01 2007-01-02 Microsoft Corporation Image processing using linear light values and other image processing improvements
US7139002B2 (en) * 2003-08-01 2006-11-21 Microsoft Corporation Bandwidth-efficient processing of video images
US7643675B2 (en) * 2003-08-01 2010-01-05 Microsoft Corporation Strategies for processing image information using a color information data structure
US7002586B2 (en) * 2003-08-29 2006-02-21 Sun Microsystems, Inc. Method and apparatus for vertex splitting in a graphics system
US20050179696A1 (en) * 2004-01-15 2005-08-18 Swallow Ronald J. Super virtual image generator
US8224639B2 (en) 2004-03-29 2012-07-17 Sony Computer Entertainment Inc. Methods and apparatus for achieving thermal management using processing task scheduling
US20060044328A1 (en) * 2004-08-26 2006-03-02 Rai Barinder S Overlay control circuit and method
US20090096798A1 (en) * 2005-01-25 2009-04-16 Reuven Bakalash Graphics Processing and Display System Employing Multiple Graphics Cores on a Silicon Chip of Monolithic Construction
US7694107B2 (en) * 2005-08-18 2010-04-06 Hewlett-Packard Development Company, L.P. Dynamic performance ratio proportionate distribution of threads with evenly divided workload by homogeneous algorithm to heterogeneous computing units
US8266232B2 (en) * 2005-10-15 2012-09-11 International Business Machines Corporation Hardware processing of commands within virtual client computing environment
JP4808477B2 (en) * 2005-11-25 2011-11-02 ザイオソフト株式会社 Image processing method and image processing program
US8125486B2 (en) * 2006-02-23 2012-02-28 Los Alamos National Security, Llc Combining multi-layered bitmap files using network specific hardware
US8775704B2 (en) 2006-04-05 2014-07-08 Nvidia Corporation Method and system for communication between a secondary processor and an auxiliary display subsystem of a notebook
EP2014032A2 (en) * 2006-04-24 2009-01-14 Koninklijke Philips Electronics N.V. Electronic device with end-to-end flow control of messages
US8487947B2 (en) * 2006-09-28 2013-07-16 Agere Systems Inc. Video processing architecture having reduced memory requirement
US8933943B2 (en) * 2008-04-30 2015-01-13 Intel Corporation Technique for performing load balancing for parallel rendering
US8736617B2 (en) * 2008-08-04 2014-05-27 Nvidia Corporation Hybrid graphic display
US8754895B2 (en) * 2008-09-09 2014-06-17 Sony Corporation Pipelined image processing engine
US9270783B2 (en) * 2008-12-06 2016-02-23 International Business Machines Corporation System and method for photorealistic imaging workload distribution
US9075559B2 (en) * 2009-02-27 2015-07-07 Nvidia Corporation Multiple graphics processing unit system and method
US20100289804A1 (en) * 2009-05-13 2010-11-18 International Business Machines Corporation System, mechanism, and apparatus for a customizable and extensible distributed rendering api
US9479358B2 (en) * 2009-05-13 2016-10-25 International Business Machines Corporation Managing graphics load balancing strategies
US9135675B2 (en) 2009-06-15 2015-09-15 Nvidia Corporation Multiple graphics processing unit display synchronization system and method
US8766989B2 (en) * 2009-07-29 2014-07-01 Nvidia Corporation Method and system for dynamically adding and removing display modes coordinated across multiple graphics processing units
US8780122B2 (en) * 2009-09-16 2014-07-15 Nvidia Corporation Techniques for transferring graphics data from system memory to a discrete GPU
US9111325B2 (en) 2009-12-31 2015-08-18 Nvidia Corporation Shared buffer techniques for heterogeneous hybrid graphics
JP5590849B2 (en) * 2009-10-08 2014-09-17 キヤノン株式会社 Data processing apparatus including parallel processing circuit having a plurality of processing modules, its control apparatus, its control method, and program
US9183608B2 (en) 2009-12-23 2015-11-10 Intel Corporation Image processing techniques for tile-based rasterization
US8593466B2 (en) * 2010-06-08 2013-11-26 Intel Corporation Tile rendering for image processing
JP5533330B2 (en) * 2010-06-23 2014-06-25 富士ゼロックス株式会社 Data processing device
AU2012258437A1 (en) 2012-11-30 2014-06-19 Canon Kabushiki Kaisha Method, apparatus and system for determining a merged intermediate representation of a page
US10003792B2 (en) 2013-05-27 2018-06-19 Microsoft Technology Licensing, Llc Video encoder for images
US9818379B2 (en) 2013-08-08 2017-11-14 Nvidia Corporation Pixel data transmission over multiple pixel interfaces
US9633409B2 (en) * 2013-08-26 2017-04-25 Apple Inc. GPU predication
CN105359531B (en) 2014-03-17 2019-08-06 微软技术许可有限责任公司 Method and system for determining for the coder side of screen content coding
US9477998B2 (en) * 2014-06-01 2016-10-25 Apple Inc. Performance control for concurrent animations
CN106416254B (en) 2015-02-06 2019-08-02 微软技术许可有限责任公司 Evaluation stage is skipped during media coding
US20160316220A1 (en) * 2015-04-21 2016-10-27 Microsoft Technology Licensing, Llc Video encoder management strategies
US10136132B2 (en) 2015-07-21 2018-11-20 Microsoft Technology Licensing, Llc Adaptive skip or zero block detection combined with transform size decision
WO2023102863A1 (en) * 2021-12-09 2023-06-15 Shanghaitech University Multi-core acceleration of neural rendering
CN116546340B (en) * 2023-07-05 2023-09-19 华中师范大学 High-speed CMOS pixel detector

Family Cites Families (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4827445A (en) 1982-02-18 1989-05-02 University Of North Carolina Image buffer having logic-enhanced pixel memory cells and method for setting values therein
US4949280A (en) * 1988-05-10 1990-08-14 Battelle Memorial Institute Parallel processor-based raster graphics system architecture
US4985834A (en) * 1988-11-22 1991-01-15 General Electric Company System and method employing pipelined parallel circuit architecture for displaying surface structures of the interior region of a solid body
EP0447228A3 (en) * 1990-03-16 1993-01-07 Hewlett-Packard Company Data stream concentrator providing attribute data storage and graphics pipeline access
US5276798A (en) * 1990-09-14 1994-01-04 Hughes Aircraft Company Multifunction high performance graphics rendering processor
US5557711A (en) * 1990-10-17 1996-09-17 Hewlett-Packard Company Apparatus and method for volume rendering
EP0575346B1 (en) * 1990-11-30 2001-10-31 Sun Microsystems, Inc. Method and apparatus for rendering graphical images
US5264837A (en) * 1991-10-31 1993-11-23 International Business Machines Corporation Video insertion processing system
GB2267203B (en) 1992-05-15 1997-03-19 Fujitsu Ltd Three-dimensional graphics drawing apparatus, and a memory apparatus to be used in texture mapping
EP0582875B1 (en) * 1992-07-27 2001-10-31 Matsushita Electric Industrial Co., Ltd. Apparatus for parallel image generation
US5394524A (en) * 1992-08-07 1995-02-28 International Business Machines Corporation Method and apparatus for processing two graphics data streams in parallel
US5388206A (en) 1992-11-13 1995-02-07 The University Of North Carolina Architecture and apparatus for image generation
JP3098342B2 (en) * 1992-11-26 2000-10-16 富士通株式会社 Processing order specification method for parallel processing equipment
US5583500A (en) 1993-02-10 1996-12-10 Ricoh Corporation Method and apparatus for parallel encoding and decoding of data
US5583974A (en) 1993-05-10 1996-12-10 Apple Computer, Inc. Computer graphics system having high performance multiple layer Z-buffer
KR940026915A (en) 1993-05-24 1994-12-10 오오가 노리오 Digital video signal recording device and playback device and recording method
US5621465A (en) 1995-04-11 1997-04-15 Matsushita Electric Industrial Co., Ltd. Color image encoder
US5999189A (en) 1995-08-04 1999-12-07 Microsoft Corporation Image compression to reduce pixel and texture memory requirements in a real-time image generator
US5990904A (en) * 1995-08-04 1999-11-23 Microsoft Corporation Method and system for merging pixel fragments in a graphics rendering system
US6009188A (en) 1996-02-16 1999-12-28 Microsoft Corporation Method and system for digital plenoptic imaging
KR100269106B1 (en) * 1996-03-21 2000-11-01 윤종용 Multiprocessor graphics system
US5870502A (en) 1996-04-08 1999-02-09 The Trustees Of Columbia University In The City Of New York System and method for a multiresolution transform of digital image information

Cited By (171)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7394462B2 (en) * 2002-05-13 2008-07-01 Kabushiki Kaisha Micronet Superimposing apparatus and method for broadcasting three-dimensional computer graphics image
US20030210351A1 (en) * 2002-05-13 2003-11-13 Kozo Murakami Superimposing apparatus and method for broadcasting three-dimensional computer graphics image
US20040054657A1 (en) * 2002-09-13 2004-03-18 Konica Minolta Holdings, Inc. Medical information management system
US7681112B1 (en) 2003-05-30 2010-03-16 Adobe Systems Incorporated Embedded reuse meta information
US7119808B2 (en) * 2003-07-15 2006-10-10 Alienware Labs Corp. Multiple parallel processor computer graphics system
US20050012749A1 (en) * 2003-07-15 2005-01-20 Nelson Gonzalez Multiple parallel processor computer graphics system
US20080211816A1 (en) * 2003-07-15 2008-09-04 Alienware Labs. Corp. Multiple parallel processor computer graphics system
US7782327B2 (en) 2003-07-15 2010-08-24 Alienware Labs. Corp. Multiple parallel processor computer graphics system
US20060290700A1 (en) * 2003-07-15 2006-12-28 Alienware Labs. Corp. Multiple parallel processor computer graphics system
US20050015718A1 (en) * 2003-07-16 2005-01-20 Sambhus Mihir Y. Method and system for client aware content aggregation and rendering in a portal server
US20050088445A1 (en) * 2003-10-22 2005-04-28 Alienware Labs Corporation Motherboard for supporting multiple graphics cards
US7782325B2 (en) 2003-10-22 2010-08-24 Alienware Labs Corporation Motherboard for supporting multiple graphics cards
US8085273B2 (en) 2003-11-19 2011-12-27 Lucid Information Technology, Ltd Multi-mode parallel graphics rendering system employing real-time automatic scene profiling and mode control
US7961194B2 (en) 2003-11-19 2011-06-14 Lucid Information Technology, Ltd. Method of controlling in real time the switching of modes of parallel operation of a multi-mode parallel graphics processing subsystem embodied within a host computing system
US7843457B2 (en) 2003-11-19 2010-11-30 Lucid Information Technology, Ltd. PC-based computing systems employing a bridge chip having a routing unit for distributing geometrical data and graphics commands to parallelized GPU-driven pipeline cores supported on a plurality of graphics cards and said bridge chip during the running of a graphics application
US7812846B2 (en) 2003-11-19 2010-10-12 Lucid Information Technology, Ltd PC-based computing system employing a silicon chip of monolithic construction having a routing unit, a control unit and a profiling unit for parallelizing the operation of multiple GPU-driven pipeline cores according to the object division mode of parallel operation
US7808499B2 (en) 2003-11-19 2010-10-05 Lucid Information Technology, Ltd. PC-based computing system employing parallelized graphics processing units (GPUS) interfaced with the central processing unit (CPU) using a PC bus and a hardware graphics hub having a router
US7800619B2 (en) 2003-11-19 2010-09-21 Lucid Information Technology, Ltd. Method of providing a PC-based computing system with parallel graphics processing capabilities
US7944450B2 (en) 2003-11-19 2011-05-17 Lucid Information Technology, Ltd. Computing system having a hybrid CPU/GPU fusion-type graphics processing pipeline (GPPL) architecture
US7800611B2 (en) 2003-11-19 2010-09-21 Lucid Information Technology, Ltd. Graphics hub subsystem for interfacing parallalized graphics processing units (GPUs) with the central processing unit (CPU) of a PC-based computing system having an CPU interface module and a PC bus
US7800610B2 (en) 2003-11-19 2010-09-21 Lucid Information Technology, Ltd. PC-based computing system employing a multi-GPU graphics pipeline architecture supporting multiple modes of GPU parallelization dymamically controlled while running a graphics application
US7796130B2 (en) 2003-11-19 2010-09-14 Lucid Information Technology, Ltd. PC-based computing system employing multiple graphics processing units (GPUS) interfaced with the central processing unit (CPU) using a PC bus and a hardware hub, and parallelized according to the object division mode of parallel operation
US7796129B2 (en) 2003-11-19 2010-09-14 Lucid Information Technology, Ltd. Multi-GPU graphics processing subsystem for installation in a PC-based computing system having a central processing unit (CPU) and a PC bus
US7940274B2 (en) 2003-11-19 2011-05-10 Lucid Information Technology, Ltd Computing system having a multiple graphics processing pipeline (GPPL) architecture supported on multiple external graphics cards connected to an integrated graphics device (IGD) embodied within a bridge circuit
US8125487B2 (en) 2003-11-19 2012-02-28 Lucid Information Technology, Ltd Game console system capable of paralleling the operation of multiple graphic processing units (GPUS) employing a graphics hub device supported on a game console board
US7777748B2 (en) 2003-11-19 2010-08-17 Lucid Information Technology, Ltd. PC-level computing system with a multi-mode parallel graphics rendering subsystem employing an automatic mode controller, responsive to performance data collected during the run-time of graphics applications
US8754894B2 (en) 2003-11-19 2014-06-17 Lucidlogix Software Solutions, Ltd. Internet-based graphics application profile management system for updating graphic application profiles stored within the multi-GPU graphics rendering subsystems of client machines running graphics-based applications
US8134563B2 (en) 2003-11-19 2012-03-13 Lucid Information Technology, Ltd Computing system having multi-mode parallel graphics rendering subsystem (MMPGRS) employing real-time automatic scene profiling and mode control
US9584592B2 (en) 2003-11-19 2017-02-28 Lucidlogix Technologies Ltd. Internet-based graphics application profile management system for updating graphic application profiles stored within the multi-GPU graphics rendering subsystems of client machines running graphics-based applications
US20080238917A1 (en) * 2003-11-19 2008-10-02 Lucid Information Technology, Ltd. Graphics hub subsystem for interfacing parallalized graphics processing units (GPUS) with the central processing unit (CPU) of a PC-based computing system having an CPU interface module and a PC bus
US20070279411A1 (en) * 2003-11-19 2007-12-06 Reuven Bakalash Method and System for Multiple 3-D Graphic Pipeline Over a Pc Bus
US20080117217A1 (en) * 2003-11-19 2008-05-22 Reuven Bakalash Multi-mode parallel graphics rendering system employing real-time automatic scene profiling and mode control
US20080117219A1 (en) * 2003-11-19 2008-05-22 Reuven Bakalash PC-based computing system employing a silicon chip of monolithic construction having a routing unit, a control unit and a profiling unit for parallelizing the operation of multiple GPU-driven pipeline cores according to the object division mode of parallel operation
US8284207B2 (en) 2003-11-19 2012-10-09 Lucid Information Technology, Ltd. Method of generating digital images of objects in 3D scenes while eliminating object overdrawing within the multiple graphics processing pipeline (GPPLS) of a parallel graphics processing system generating partial color-based complementary-type images along the viewing direction using black pixel rendering and subsequent recompositing operations
US8754897B2 (en) 2004-01-28 2014-06-17 Lucidlogix Software Solutions, Ltd. Silicon chip of a monolithic construction for use in implementing multiple graphic cores in a graphics processing and display subsystem
US9659340B2 (en) 2004-01-28 2017-05-23 Lucidlogix Technologies Ltd Silicon chip of a monolithic construction for use in implementing multiple graphic cores in a graphics processing and display subsystem
US20080129745A1 (en) * 2004-01-28 2008-06-05 Lucid Information Technology, Ltd. Graphics subsytem for integation in a PC-based computing system and providing multiple GPU-driven pipeline cores supporting multiple modes of parallelization dynamically controlled while running a graphics application
US20060232590A1 (en) * 2004-01-28 2006-10-19 Reuven Bakalash Graphics processing and display system employing multiple graphics cores on a silicon chip of monolithic construction
US7834880B2 (en) 2004-01-28 2010-11-16 Lucid Information Technology, Ltd. Graphics processing and display system employing multiple graphics cores on a silicon chip of monolithic construction
US20080129744A1 (en) * 2004-01-28 2008-06-05 Lucid Information Technology, Ltd. PC-based computing system employing a silicon chip implementing parallelized GPU-driven pipelines cores supporting multiple modes of parallelization dynamically controlled while running a graphics application
US7812845B2 (en) 2004-01-28 2010-10-12 Lucid Information Technology, Ltd. PC-based computing system employing a silicon chip implementing parallelized GPU-driven pipelines cores supporting multiple modes of parallelization dynamically controlled while running a graphics application
US20060279577A1 (en) * 2004-01-28 2006-12-14 Reuven Bakalash Graphics processing and display system employing multiple graphics cores on a silicon chip of monolithic construction
US7812844B2 (en) 2004-01-28 2010-10-12 Lucid Information Technology, Ltd. PC-based computing system employing a silicon chip having a routing unit and a control unit for parallelizing multiple GPU-driven pipeline cores according to the object division mode of parallel operation during the running of a graphics application
US7808504B2 (en) 2004-01-28 2010-10-05 Lucid Information Technology, Ltd. PC-based computing system having an integrated graphics subsystem supporting parallel graphics processing operations across a plurality of different graphics processing units (GPUS) from the same or different vendors, in a manner transparent to graphics applications
US8040353B2 (en) 2004-04-16 2011-10-18 Apple Inc. System for emulating graphics operations
US10402934B2 (en) 2004-04-16 2019-09-03 Apple Inc. System for optimizing graphics operations
US20050231514A1 (en) * 2004-04-16 2005-10-20 John Harper System for optimizing graphics operations
US20070247468A1 (en) * 2004-04-16 2007-10-25 Mark Zimmer System and method for processing graphics operations with graphics processing unit
US8520021B2 (en) 2004-04-16 2013-08-27 Apple Inc. System and method for processing graphics operations with graphics processing unit
US20110037768A1 (en) * 2004-04-16 2011-02-17 Apple Inc. System for Emulating Graphics Operations
US20050231521A1 (en) * 2004-04-16 2005-10-20 John Harper System for reducing the number of programs necessary to render an image
US7667709B2 (en) 2004-04-16 2010-02-23 Apple Inc. System and method for processing graphics operations with graphics processing unit
US8134561B2 (en) 2004-04-16 2012-03-13 Apple Inc. System for optimizing graphics operations
US20050235287A1 (en) * 2004-04-16 2005-10-20 John Harper System for emulating graphics operations
US8044963B2 (en) 2004-04-16 2011-10-25 Apple Inc. System for emulating graphics operations
US8040359B2 (en) 2004-04-16 2011-10-18 Apple Inc. System for emulating graphics operations
US20070182747A1 (en) * 2004-04-16 2007-08-09 John Harper High-level program interface for graphics operations
US8446416B2 (en) 2004-04-16 2013-05-21 Apple Inc. System for optimizing graphics operations
US8009176B2 (en) 2004-04-16 2011-08-30 Apple Inc. System and method for processing graphics operations with graphics processing unit
US20110187736A1 (en) * 2004-04-16 2011-08-04 Apple Inc. System and Method for Processing Graphics Operations with Graphics Processing Unit
US20110169857A1 (en) * 2004-04-16 2011-07-14 Apple Inc. System for Optimizing Graphics Operations
US20070174827A1 (en) * 2004-04-16 2007-07-26 John Harper System for reducing the number of programs necessary to render an image
US20100214305A1 (en) * 2004-04-16 2010-08-26 Apple Inc. System and Method for Processing Graphics Operations with Graphics Processing Unit
US7788656B2 (en) 2004-04-16 2010-08-31 Apple Inc. System for reducing the number of programs necessary to render an image
US8704837B2 (en) 2004-04-16 2014-04-22 Apple Inc. High-level program interface for graphics operations
US20060125839A1 (en) * 2004-04-16 2006-06-15 John Harper System for reducing the number of programs necessary to render an image
US7911472B2 (en) 2004-04-16 2011-03-22 Apple Inc. System for reducing the number of programs necessary to render an image
US7231632B2 (en) 2004-04-16 2007-06-12 Apple Computer, Inc. System for reducing the number of programs necessary to render an image
US20110074810A1 (en) * 2004-04-16 2011-03-31 Apple Inc. System for Emulating Graphics Operations
US20110074821A1 (en) * 2004-04-16 2011-03-31 Apple Inc. System for Emulating Graphics Operations
US20060125838A1 (en) * 2004-04-16 2006-06-15 John Harper System for reducing the number of programs necessary to render an image
US9691118B2 (en) 2004-04-16 2017-06-27 Apple Inc. System for optimizing graphics operations
US7614041B2 (en) 2004-04-16 2009-11-03 Apple Inc. System for reducing the number of programs necessary to render an image
US7847800B2 (en) 2004-04-16 2010-12-07 Apple Inc. System for emulating graphics operations
US20160360381A1 (en) * 2004-04-30 2016-12-08 Unify Gmbh & Co. Kg Method and Unit Used to Determine Useable Services
US9467562B2 (en) * 2004-04-30 2016-10-11 Unify Gmbh & Co. Kg Method and unit used to determine useable services
US20070242819A1 (en) * 2004-04-30 2007-10-18 Siemens Aktiengesellschaft Method and Unit Used to Determine Useable Services
US10455383B2 (en) * 2004-04-30 2019-10-22 Unify Gmbh & Co. Kg Method and unit used to determine useable services
US20070171233A1 (en) * 2004-06-24 2007-07-26 Mark Zimmer Resolution independent user interface design
US20070180391A1 (en) * 2004-06-24 2007-08-02 Apple Computer, Inc. User-interface design
US7907146B2 (en) 2004-06-24 2011-03-15 Apple Inc. Resolution independent user interface design
US8508549B2 (en) 2004-06-24 2013-08-13 Apple Inc. User-interface design
US20060284878A1 (en) * 2004-06-24 2006-12-21 Apple Computer, Inc. Resolution Independent User Interface Design
US8130224B2 (en) 2004-06-24 2012-03-06 Apple Inc. User-interface design
US8130237B2 (en) 2004-06-24 2012-03-06 Apple Inc. Resolution independent user interface design
US20050285965A1 (en) * 2004-06-24 2005-12-29 Apple Computer, Inc. User-interface design
US8068103B2 (en) 2004-06-24 2011-11-29 Apple Inc. User-interface design
US8453065B2 (en) 2004-06-25 2013-05-28 Apple Inc. Preview and installation of user interface elements in a display environment
US8144159B2 (en) 2004-06-25 2012-03-27 Apple Inc. Partial display updates in a windowing system using a programmable graphics processing unit
US7969453B2 (en) 2004-06-25 2011-06-28 Apple Inc. Partial display updates in a windowing system using a programmable graphics processing unit
US7793232B2 (en) 2004-06-25 2010-09-07 Apple Inc. Unified interest layer for user interface
US7793222B2 (en) 2004-06-25 2010-09-07 Apple Inc. User interface element with auxiliary function
US20070182749A1 (en) * 2004-06-25 2007-08-09 Apple Computer, Inc. Partial display updates in a windowing system using a programmable graphics processing unit
US7761800B2 (en) 2004-06-25 2010-07-20 Apple Inc. Unified interest layer for user interface
US20110216079A1 (en) * 2004-06-25 2011-09-08 Apple Inc. Partial Display Updates in a Windowing System Using a Programmable Graphics Processing Unit
US8302020B2 (en) 2004-06-25 2012-10-30 Apple Inc. Widget authoring and editing environment
US8291332B2 (en) 2004-06-25 2012-10-16 Apple Inc. Layer for accessing user interface elements
US7873910B2 (en) 2004-06-25 2011-01-18 Apple Inc. Configuration bar for lauching layer for accessing user interface elements
US7984384B2 (en) 2004-06-25 2011-07-19 Apple Inc. Web view layer for accessing user interface elements
US8566732B2 (en) 2004-06-25 2013-10-22 Apple Inc. Synchronization of widgets and dashboards
US20050285867A1 (en) * 2004-06-25 2005-12-29 Apple Computer, Inc. Partial display updates in a windowing system using a programmable graphics processing unit
US20070257925A1 (en) * 2004-06-25 2007-11-08 Apple Computer, Inc. Partial display updates in a windowing system using a programmable graphics processing unit
US9477646B2 (en) 2004-06-25 2016-10-25 Apple Inc. Procedurally expressing graphic objects for web pages
US10489040B2 (en) 2004-06-25 2019-11-26 Apple Inc. Visual characteristics of user interface elements in a unified interest layer
US8239749B2 (en) 2004-06-25 2012-08-07 Apple Inc. Procedurally expressing graphic objects for web pages
US10387549B2 (en) 2004-06-25 2019-08-20 Apple Inc. Procedurally expressing graphic objects for web pages
US9507503B2 (en) 2004-06-25 2016-11-29 Apple Inc. Remote access to layer and user interface elements
US9753627B2 (en) 2004-06-25 2017-09-05 Apple Inc. Visual characteristics of user interface elements in a unified interest layer
US7652678B2 (en) 2004-06-25 2010-01-26 Apple Inc. Partial display updates in a windowing system using a programmable graphics processing unit
US20050285866A1 (en) * 2004-06-25 2005-12-29 Apple Computer, Inc. Display-wide visual effects for a windowing system using a programmable graphics processing unit
US8266538B2 (en) 2004-06-25 2012-09-11 Apple Inc. Remote access to layer and user interface elements
US7542042B1 (en) * 2004-11-10 2009-06-02 Nvidia Corporation Subpicture overlay using fragment shader
US8140975B2 (en) 2005-01-07 2012-03-20 Apple Inc. Slide show navigation
US9384470B2 (en) 2005-01-07 2016-07-05 Apple Inc. Slide show navigation
US10867364B2 (en) 2005-01-25 2020-12-15 Google Llc System on chip having processing and graphics units
US11341602B2 (en) 2005-01-25 2022-05-24 Google Llc System on chip having processing and graphics units
US10614545B2 (en) 2005-01-25 2020-04-07 Google Llc System on chip having processing and graphics units
US8543931B2 (en) 2005-06-07 2013-09-24 Apple Inc. Preview including theme based installation of user interface elements in a display environment
US8073225B2 (en) * 2005-06-10 2011-12-06 Siemens Aktiengesellschaft Method and device for controlling image data acquisition and image reconstruction
US20060285735A1 (en) * 2005-06-10 2006-12-21 Ulrich Hagen Method and device for controlling image data acquisition and image reconstruction
US20070035559A1 (en) * 2005-08-12 2007-02-15 Microsoft Corporation Compositing external images into a multimedia rendering pipeline
US7477264B2 (en) * 2005-08-12 2009-01-13 Microsoft Corporation Compositing external images into a multimedia rendering pipeline
US20070076007A1 (en) * 2005-09-21 2007-04-05 Quanta Computer Inc. Display controller capable of reducing cache memory and the frame adjusting method thereof
US9513930B2 (en) 2005-10-27 2016-12-06 Apple Inc. Workflow widgets
US8543824B2 (en) 2005-10-27 2013-09-24 Apple Inc. Safe distribution and use of content
US11150781B2 (en) 2005-10-27 2021-10-19 Apple Inc. Workflow widgets
US9104294B2 (en) 2005-10-27 2015-08-11 Apple Inc. Linked widgets
US7954064B2 (en) 2005-10-27 2011-05-31 Apple Inc. Multiple dashboards
US9032318B2 (en) 2005-10-27 2015-05-12 Apple Inc. Widget security
US7752556B2 (en) 2005-10-27 2010-07-06 Apple Inc. Workflow widgets
US7743336B2 (en) 2005-10-27 2010-06-22 Apple Inc. Widget security
US9417888B2 (en) 2005-11-18 2016-08-16 Apple Inc. Management of user interface elements in a display environment
US7707514B2 (en) 2005-11-18 2010-04-27 Apple Inc. Management of user interface elements in a display environment
US8869027B2 (en) 2006-08-04 2014-10-21 Apple Inc. Management and generation of dashboards
US20100033502A1 (en) * 2006-10-13 2010-02-11 Freescale Semiconductor, Inc. Image processing apparatus for superimposing windows displaying video data having different frame rates
US20080158236A1 (en) * 2006-12-31 2008-07-03 Reuven Bakalash Parallel graphics system employing multiple graphics pipelines wtih multiple graphics processing units (GPUs) and supporting the object division mode of parallel graphics rendering using pixel processing resources provided therewithin
US8497865B2 (en) 2006-12-31 2013-07-30 Lucid Information Technology, Ltd. Parallel graphics system employing multiple graphics processing pipelines with multiple graphics processing units (GPUS) and supporting an object division mode of parallel graphics processing using programmable pixel or vertex processing resources provided with the GPUS
US8638328B2 (en) * 2007-01-05 2014-01-28 Landmark Graphics Corporation Systems and methods for visualizing multiple volumetric data sets in real time
US20080165186A1 (en) * 2007-01-05 2008-07-10 Landmark Graphics Corporation, A Halliburton Company Systems and methods for visualizing multiple volumetric data sets in real time
US20080165190A1 (en) * 2007-01-09 2008-07-10 Samsung Electronics Co., Ltd. Apparatus and method of displaying overlaid image
US20080250227A1 (en) * 2007-04-04 2008-10-09 Linderman Michael D General Purpose Multiprocessor Programming Apparatus And Method
US9483164B2 (en) 2007-07-18 2016-11-01 Apple Inc. User-centric widgets and dashboards
US8954871B2 (en) 2007-07-18 2015-02-10 Apple Inc. User-centric widgets and dashboards
US8667415B2 (en) 2007-08-06 2014-03-04 Apple Inc. Web widgets
US8156467B2 (en) 2007-08-27 2012-04-10 Adobe Systems Incorporated Reusing components in a running application
US8176466B2 (en) 2007-10-01 2012-05-08 Adobe Systems Incorporated System and method for generating an application fragment
US20090106530A1 (en) * 2007-10-17 2009-04-23 Christian Lauterbach System, method, and computer program product for generating a ray tracing data structure utilizing a parallel processor architecture
US8072460B2 (en) * 2007-10-17 2011-12-06 Nvidia Corporation System, method, and computer program product for generating a ray tracing data structure utilizing a parallel processor architecture
US9619304B2 (en) 2008-02-05 2017-04-11 Adobe Systems Incorporated Automatic connections between application components
US8736600B2 (en) 2008-06-06 2014-05-27 Landmark Graphics Corporation Systems and methods for imaging a three-dimensional volume of geometrically irregular grid data representing a grid volume
US20090303233A1 (en) * 2008-06-06 2009-12-10 Landmark Graphics Corporation, A Halliburton Company Systems and Methods for Imaging a Three-Dimensional Volume of Geometrically Irregular Grid Data Representing a Grid Volume
US9870629B2 (en) 2008-06-20 2018-01-16 New Bis Safe Luxco S.À R.L Methods, apparatus and systems for data visualization and related applications
WO2009154484A3 (en) * 2008-06-20 2010-02-11 Business Intelligence Solutions Safe B.V. Methods, apparatus and systems for data visualization and related applications
WO2009154484A2 (en) * 2008-06-20 2009-12-23 Business Intelligence Solutions Safe B.V. Methods, apparatus and systems for data visualization and related applications
US8656293B1 (en) 2008-07-29 2014-02-18 Adobe Systems Incorporated Configuring mobile devices
US8537232B2 (en) * 2008-10-08 2013-09-17 Fuji Xerox Co., Ltd. Image processing apparatus, image processing system and computer readable medium
US20100085441A1 (en) * 2008-10-08 2010-04-08 Fuji Xerox Co., Ltd. Image processing apparatus, image processing system and computer readable medium
US20100149195A1 (en) * 2008-12-15 2010-06-17 Canon Kabushiki Kaisha Load balancing in multiple processor rendering systems
US8704838B2 (en) 2008-12-15 2014-04-22 Canon Kabushiki Kaisha Load balancing in multiple processor rendering systems
US9311739B2 (en) * 2009-05-28 2016-04-12 Siliconarts, Inc. Ray tracing core and ray tracing chip having the same
US9965889B2 (en) 2009-05-28 2018-05-08 Siliconarts, Inc. Ray tracing core and ray tracing chip having the same
US20120069023A1 (en) * 2009-05-28 2012-03-22 Siliconarts, Inc. Ray tracing core and ray tracing chip having the same
US20130100498A1 (en) * 2011-10-25 2013-04-25 Canon Kabushiki Kaisha Image formation processing apparatus and image processing method
US8824010B2 (en) * 2011-10-25 2014-09-02 Canon Kabushiki Kaisha Image formation processing apparatus and image processing method
US20140168231A1 (en) * 2012-12-18 2014-06-19 Nvidia Corporation Triggering performance event capture via pipelined state bundles
US9030480B2 (en) * 2012-12-18 2015-05-12 Nvidia Corporation Triggering performance event capture via pipelined state bundles
US9799091B2 (en) * 2014-11-20 2017-10-24 Intel Corporation Apparatus and method for efficient frame-to-frame coherency exploitation for sort-last architectures
US20160148339A1 (en) * 2014-11-20 2016-05-26 Intel Corporation Apparatus and method for efficient frame-to-frame coherency exploitation for sort-last architectures
US10452581B2 (en) 2015-01-19 2019-10-22 Microsoft Technology Licensing, Llc Memory descriptor list caching and pipeline processing
US20160210159A1 (en) * 2015-01-19 2016-07-21 Microsoft Microsoft Technology Licensing, LLC User Mode Driver Extension and Preprocessing
US9817776B2 (en) 2015-01-19 2017-11-14 Microsoft Technology Licensing, Llc Memory descriptor list caching and pipeline processing

Also Published As

Publication number Publication date
US6753878B1 (en) 2004-06-22

Similar Documents

Publication Publication Date Title
US6753878B1 (en) Parallel pipelined merge engines
US9405586B2 (en) Method of dynamic load-balancing within a PC-based computing system employing a multiple GPU-based graphics pipeline architecture supporting multiple modes of GPU parallelization
US9584592B2 (en) Internet-based graphics application profile management system for updating graphic application profiles stored within the multi-GPU graphics rendering subsystems of client machines running graphics-based applications
US7522171B1 (en) On-the-fly reordering of 32-bit per component texture images in a multi-cycle data transfer
US8237705B2 (en) Hierarchical processor array
US7071935B1 (en) Graphics system with just-in-time decompression of compressed graphics data
US5821949A (en) Three-dimensional graphics accelerator with direct data channels for improved performance
US7777748B2 (en) PC-level computing system with a multi-mode parallel graphics rendering subsystem employing an automatic mode controller, responsive to performance data collected during the run-time of graphics applications
US6628277B1 (en) Decompression of three-dimensional graphics data using mesh buffer references to reduce redundancy of processing
Moll et al. Sepia: scalable 3D compositing using PCI Pamette
US20080094403A1 (en) Computing system capable of parallelizing the operation graphics processing units (GPUs) supported on a CPU/GPU fusion-architecture chip and one or more external graphics cards, employing a software-implemented multi-mode parallel graphics rendering subsystem
JP2005521178A (en) Scalable high-performance 3D graphics
US7484076B1 (en) Executing an SIMD instruction requiring P operations on an execution unit that performs Q operations at a time (Q&lt;P)
CN116188241A (en) Graphics processor, method of operation, and machine-readable storage medium
US7489315B1 (en) Pixel stream assembly for raster operations
Hinkenjann et al. Mixed-mode parallel real-time rendering on commodity hardware
Williams et al. Sort-middle multi-projector immediate-mode rendering in chromium
Santilli Vortex: deferred sort last parallel graphics architecture
Huang et al. Design a PC-CAVE environment using the parallel rendering approach

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION