KR20010113669A - A graphics system configured to perform parallel sample to pixel calculation - Google Patents

Abstract

A computer graphics system configured to utilize a sample buffer 162 and a plurality of parallel sample-pixel operation units 170, wherein the sample-pixel operation unit is configured to access different portions of the sample buffer in parallel. The graphics system may include a processor 352, a sample buffer 162, and a plurality of sample-pixel computing units 170, 360. The processor is configured to receive a set of graphical data and render samples based on the graphical data. The sample buffer is configured to store the samples. The sample-pixel calculation unit reads and filters the samples from the sample buffer to produce an output pixel. Each of the sample-pixel calculation units is configured to generate pixels corresponding to different regions of the image. The area may be a vertical or horizontal stripe of the image or a rectangular portion of the image. The regions may overlap to prevent visual aberration.

Description

A GRAPHICS SYSTEM CONFIGURED TO PERFORM PARALLEL SAMPLE TO PIXEL CALCULATION}

Computer systems typically rely on graphics systems to produce visual output on a computer screen or display device. Early graphics systems were only responsible for taking the output from the processor and displaying it on the screen. In essence, they acted as simple repeaters or interfaces. However, modern graphics systems include graphics processors with great processing power. It acts as a coprocessor rather than just a repeater. This change is due to an increase in the complexity and amount of data recently sent to display devices. Modern computer displays, for example, have more pixels and greater color depth and can display more complex images at faster refresh rates than earlier models. Likewise, the image to be represented is more complex and may include advanced techniques such as anti-aliasing and texture mapping.

As a result, without significant processing power in the graphics system, the CPU will spend a tremendous amount of time performing graphics calculations. This deprives the computer system of the processing power needed to execute other tasks related to program execution, greatly reducing overall system performance. In a powerful graphics system, however, if the CPU is instructed to draw a box on the screen, the CPU does not have to calculate the position and color of each pixel. Instead, the CPU can request the video card to "draw a box at this coordinate." The graphics system then draws a box, allowing the processor to freely perform other tasks.

In general, a graphics system in a computer (also simply called a graphics system) is a type of video adapter that includes its own processor that amplifies the performance level. Since these processors are specialized for calculating graphics transformations, they tend to achieve better results than the general purpose CPUs used in computer systems. Moreover, they allow the CPU to freely execute other instructions while the graphics system handles graphics computation. The popularity of graphics applications, and in particular multimedia applications, has made high-performance graphics systems a common computer system. Most computer manufacturers now include high performance graphics systems in their systems.

Since graphics systems typically perform only a limited set of functions, they can be customized to order and therefore much more efficient in graphics than the general purpose central processor of a computer. While early graphics systems were limited to performing two-dimensional (2D) graphics, their functionality increased to support three-dimensional (3D) wire-frame graphics, 3D stereoscopic, and now advanced shading and fogging Includes support for three-dimensional (3D) graphics with textures and special effects such as fogging, alpha-blending, and specular highlighting.

The processing power of 3D graphics systems has advanced at a steep pace. A few years ago, a shaded image of a simple object could only render at a few frames per second, while today's systems support rendering complex objects at frequencies of 60 Hz and above. In this increase in speed, in the not-so-distant future, the graphics system will literally be able to produce more pixels than a single human visual system can recognize. While this particular capability can be used in a multi-viewer environment, it can be wasted in a more general basic single-viewer environment. Accordingly, there is a need for a graphics system that has the ability to match the various properties of human resolution tissue (ie, the ability to have the required quality or the most perceived quality).

The number of pixels is an important factor in graphics system performance, but another important factor is the quality of the image. For example, if the edges of an image are too sharp or jagged (sometimes called "aliased"), even high pixel density images can still look unrealistic. A well known technique for overcoming this problem is anti-aliasing. Anti-aliasing involves smoothing the edges of an object by shading pixels along the boundaries of graphical elements. More specifically, anti-aliasing involves removing high frequency components from an image before the high frequency components disturb visual artifacts. For example, anti-aliasing can cause certain pixels to have a median (eg, around the shadow of a bright object overlaid with a dark background) to soften or smooth the edges of high contrast in the image. have.

Another visual effect used to increase the reality of computer images is alpha blending. Alpha blending is a technique for controlling the transparency of objects, enabling realistic rendering of transparent surfaces such as water and glass. Another effect used to further increase realism is fogging. Fogging blurs objects as they move away from the viewer. Simple fogging is a special case of alpha mixing, where the degree of alpha changes with distance, making it appear as fog as the object moves away from the viewer. This simple fogging is also called "depth cueing" or atmospheric attenuation, which lowers the contrast of the object, making it less visible as the object retreats. More complex types of fogging go beyond simple linear functions to provide a more complex relationship between transparency level and distance from the viewer. The current state of the art software systems is further advanced by utilizing atmospheric models to provide improved realism in the underlying fog.

While the techniques listed above dramatically improve the appearance of computer graphics images, they also have certain limitations. In particular, they can exhibit their own aberrations and are typically limited by the pixel density represented in the display device.

As a result, there is a need for a graphics system that can utilize improved performance levels to improve the quality of the rendered image as well as the number of pixels rendered. Moreover, a graphics system that can improve processing power, such as anti-aliasing, that enhances graphics effects is desirable.

Prior art graphics systems generally fall short of these purposes. Prior art graphics systems use conventional frame buffers for refreshing pixels / video on a display. The frame buffer stores the pixels of the rows and columns that exactly match each row and column on the display. Prior art graphics systems render 2D and / or 3D images or objects in the form of pixels into the frame buffer, and then read the pixels from the frame buffer during screen refresh to refresh the display. Thus, the frame buffer stores the output pixels provided to the display. In order to reduce the visual artifacts that can occur as the frame buffer is refreshed and the screen is refreshed, the frame buffer of most graphics systems is a double buffer.

In order to obtain more realistic images, several prior art graphics systems have been advanced by generating two or more samples per pixel. As used herein, the term "sample" refers to calculated color information that suggests the color, color depth z, transparency, and potentially other information of a particular location on an object or image. For example, one sample may include the following component values: a red value, a green value, a blue value, a z value, and an alpha value (eg, indicating the transparency of the sample). One sample may also include other information such as, for example, z-depth value, blurring value, intensity value, relative brightness information, indicating that the sample is composed of partial or complete control information rather than color information. That is, "sample control information"). By calculating more samples than pixels (ie, super-sampling), more detailed images can be calculated than can be represented on a display device. For example, the graphics system can calculate four samples for each pixel that can be output to the display device. After the samples have been calculated, they are synthesized or filtered to form pixels that are stored in the frame buffer and sent to a display device. Pixels formed in this way can be used to create a more realistic final image, since even abrupt changes in the image can be smoothed by the filtering process.

These prior art super-sampling systems typically produce a much larger number of samples than the number of pixel locations on the display. These prior art systems typically have a rendering processor that calculates a sample and stores it in a render buffer. The filtering hardware then reads the samples from the render buffer, filters the samples to make a pixel, and then stores the pixels in a traditional frame buffer. Traditional frame buffers are typically double-buffers, one of which is used to refresh the display device while one is updated by the filtering hardware. Once the sample is filtered, the resulting pixels are stored in a traditional frame buffer used to refresh the display device. However, these systems have generally suffered from the limitations imposed by traditional frame buffers and the additional latency imposed by render buffers and filtration. Therefore, there is a need for an advanced graphics system that includes the benefits of pixel super-sampling while avoiding the drawbacks of traditional frame buffers.

US Patent Application No. 09/251844, entitled "Graphics System With a Variable Resolution Sample," describes a super-sampled sample buffer that is super-sampled to refresh the display; Represents a computer graphics system utilizing a sample-pixel calculation unit. The graphics processor generates a plurality of samples and stores them in a sample buffer. The graphics processor preferably generates and stores two or more samples for at least one portion of the pixel locations on the display. Thus, the sample buffer is a super-sampled sample buffer that stores samples that can be much larger than the number of pixel locations on the display. The sample-pixel calculation unit is configured to read samples from a super-sampled sample buffer, filter the samples into individual output pixels, or convolve. , Where the output pixels are provided to refresh the display. The sample-pixel calculation unit selects one or more samples and filters them to produce an output pixel. The sample-pixel computing unit may be operable to obtain a sample and generate a pixel that is provided directly to the display without intervening the frame buffer.

FIELD OF THE INVENTION The present invention generally relates to the field of computer graphics, and more particularly to high performance graphics systems.

Other objects, features and advantages of the present invention as well as the above-described objects will be more fully understood by referring to the following detailed description in conjunction with the accompanying drawings.

1A is an illustration of a computer system including an example of a graphics system;

1B illustrates another embodiment of a computer system including a graphics system.

1C illustrates another embodiment of a computer system that is part of a virtual reality workstation.

2 shows an example of a network to which the computer system of FIGS. 1A-C is connected;

3A is a diagram illustrating another embodiment of the graphics system of FIG. 1 as a virtual reality workstation.

3B is a more detailed diagram illustrating an example of a graphics system with a sample buffer;

4 shows a conventional pixel calculation;

5A shows an example of super-sampling;

5B shows random distribution of samples;

FIG. 6 illustrates in detail an example of a graphics system having an example of a variable resolution super-sampled sample buffer;

7 details another example of a graphics system having an example of a variable resolution super-sampled sample buffer and a sample-position memory in double-buffer;

8 details an embodiment of three different sample location diagrams;

9 shows an example of a sample position diagram in detail;

10 shows another example of a sample position diagram in detail;

11A illustrates in detail an example of a graphics system configured to convert samples in parallel to pixels using vertical screen rows (columns);

FIG. 11B details another example of a graphics system configured to convert samples in parallel to pixels using vertical screen rows (columns); FIG.

12 illustrates in detail another example of a graphics system configured to convert samples in parallel to pixels using horizontal screen lines (rows);

13 illustrates in detail another example of a graphics system configured to convert samples in parallel to pixels using rectangular regions;

14 is a detailed illustration of a method of reading a sample from a sample buffer;

15 illustrates in detail an example of a method for dealing with boundary conditions;

16 is a detailed view of another example of a method of dealing with a boundary condition;

17 is a flow chart illustrating an example of a method of drawing a sample into a sample buffer being super-sampled;

18 illustrates an example of a method of coding triangle vertices;

19 illustrates an example of a method of computing pixels from a sample;

20 illustrates in detail an example of computing samples in pixels, for an example sample set;

21 shows an example of a method of varying the density of a sample;

22 shows another example of a method of varying the density of a sample;

23 shows another example of one method of varying the density of a sample;

24A-B detail an example of a method of utilizing eye-tracking to change the density of a sample;

25A-B illustrate an example of a method of utilizing eye-tracking to change the density of a sample.

While the invention is capable of various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described in detail below. However, the drawings and detailed description therein are not intended to limit the invention to the precise forms disclosed, and on the contrary, all modifications, equivalents, within the spirit and scope of the invention as defined by the appended claims. , Including replacement.

The problem presented above is a graphics system configured to utilize a sample buffer and a plurality of parallel sample-pixel computation units, at least in part by a graphics system configured to access the different portions of the sample buffer in parallel. Could be solved. Advantageously, such a configuration may (depending on the embodiment) allow the graphics system to use a sample buffer instead of a traditional frame buffer that stores pixels. Since the sample-pixel computing unit can be configured to operate in parallel, the latency of the graphics system may be reduced in some embodiments.

In one example, the graphics system may include one or more graphics processors, a sample buffer and a plurality of sample-pixel computing units. The graphics processor may receive a set of three-dimensional graphic data and render a plurality of samples based on the graphic data. The sample buffer may be configured to store the plurality of samples for the sample-pixel operation unit (eg, in a dual-buffer configuration), the sample-pixel operation unit receiving and filtering samples from the sample buffer It can be configured to generate an output pixel. The output pixel can be used to form an image on the display device. Each sample-pixel calculation unit is configured to generate pixels corresponding to different regions of the image. The region may be a vertical line (ie, a column), a horizontal line (ie, a row) of the image, or a rectangular shaped portion of the image. Note that the terms "horizontal row" and "horizontal line" as used herein may be used interchangeably, and "vertical column" and "vertical line" may be used interchangeably. Each region may overlap other regions of the image to prevent visual aberrations (eg, gold, lines, gaps in the image). As mentioned above, each sample-pixel calculation unit can be advantageously configured to operate in parallel in its own area or in several areas. The sample-pixel computation unit determines (i) which samples are within a predetermined filter envelope, (ii) multiplies the samples by weighting, (iii) adds the resulting value, and (iv) The samples are configured to process the samples by normalizing the result to form an output pixel. The weight value may change depending on the position of the sample within the filter envelope (eg, the weight will decrease as the sample moves away from the center of the filter envelope). In some embodiments, the weighting factor may be normalized or pre-normalized, in which case the output will be normalized so that the resulting output pixels will not be normalized. The normalized weighting factor is adjusted to ensure that pixels generated with fewer contribution samples do not overpower pixels created with more contribution samples. Normalization may generally be performed in embodiments of a graphics system in which a variable number of samples may contribute to each output pixel. Normalization may also be performed in systems that allow for variable sample patterns and in systems where the pitch of the center of the filter varies widely with respect to the sample pattern.

In some embodiments, the graphics system may be configured to flexibly change the size or type of area used (eg, vary the width of the vertical columns used in the frame reference). Some embodiments of the graphics system may support variable resolution or variable density frame buffers. In these configurations, the graphics system can render the sample more densely in any particular area of the image (eg, the center of the image or the portion of the image in which the viewer's attention is most likely to be focused). Advantageously, the ability to flexibly change the size and / or appearance of the area used allows the graphics system to uniformize (or even uniformly) the number of samples each sample-pixel computing unit processes for a particular frame. Close).

The sample may comprise a color component and an alpha (eg transparent) component and may be stored in “bins” to simplify the process of storing and retrieving the sample from the sample buffer. As described in more detail below, bins are a means of organizing and dividing a sample buffer into a smaller set of storage locations. Moreover, in some embodiments three-dimensional graphic data may be received in compressed form (eg, geometric compression). In these embodiments the graphics processor may be configured to decompress the three-dimensional graphics data before rendering the sample. As used herein, the term "color component" includes information on a per-sample or per-pixel basis used to determine the color of a pixel or sample. For example, RGB information and transparent information may be color components.

Methods of rendering a set of three-dimensional graphic data may also be considered. In one example the method comprises (i) receiving three-dimensional graphic data, (ii) generating one or more samples based on the graphic data, (iii) storing the sample, (iv) selecting the stored sample, (iv) filtering the selected samples in parallel to produce an output pixel. The stored sample may be selected according to a plurality of regions, as described above.

Computer System-Figure 1

Referring now to FIG. 1A, an example of a computer system 80 including a three-dimensional (3-D) graphics system is shown. 3-D graphics systems include, among others, computer systems, network PCs, Internet devices, televisions including HDTV systems and interactive television systems, personal digital assistants (PDAs), and other devices that display 2D and / or 3D graphics. It can be included in either.

As shown, computer system 80 includes system unit 82 and a video monitor or display device 84 coupled to system unit 82. Display device 84 may be any of a variety of types of display monitors or devices (eg, CRTs, LCDs, gas-plasma displays, digital mirror displays (DMDs) or reflective silicon LCDs, etc.). Keyboard 86 and / or mouse 88, or other input device (e.g., trackball, digitizer, tablet, 6-degree free input device, head tracker, eye tracker, data glove) various input devices, including glove, body sensor, etc., may be coupled to the computer system. Application software may be executed by the computer system 80 to present a 3-D graphical object on the display device 84.

Computer system 80 may also include eye-tracking sensor 92 and / or 3D-glasses 90. The 3D glasses 90 may be active (eg, LCD shutter type) or passive (eg, polarized, red-green, etc.), allowing the user to view more three-dimensional images on the display device 84. I can do it. With glasses 90, each eye receives a slightly different image, and the viewer's mind interprets it as a "true" three-dimensional figure. The sensor 92 may be configured to determine which part of the image on the display device 84 the viewer is looking at (ie, the viewer's field of view is centered). The information provided from the sensor 92 can be used in a number of different ways, as described below.

Virtual Reality Computer System-Fig 1B

1B illustrates another example of a computer system 70. In this embodiment, the system includes a head mounted display device 72, a head-tracking sensor 74, and a data glove 76. The head mounted display 72 may be coupled to the system unit 82 via one or more of optical fiber connections 94 or electrically conductive connections, infrared connections, or wireless (eg, radio frequency) connections.

Computer Network-Figure 2

Referring now to FIG. 2, a computer network 500 is shown that includes at least one server computer 502 and one or more client computers 506A-N. (In the embodiment shown in FIG. 4, client computers 506A-B are depicted). One or more client systems may be configured similarly to computer system 80, each having one or more graphics systems 112 as described above. Server 502 and client (s) 506 may be connected through various connections 504, such as a local area network (LAN), wide area network (WAN), or an Internet connection. In one example, server 502 can store and forward 3-D geometric data (which may be compressed) to client 506. Client 506 receives the compressed 3-D geometric data, decompresses (if necessary), and renders the geometric data. The rendered image is then rendered on the client's display device. The client renders the geometric data and renders the image using the super-sampled sample buffer and the real-time filter technique described above. In another embodiment, compressed 3-D geometric data may be passed between client computers 506.

Computer System Block Diagram-Figure 3A

3A shows a simplified block diagram illustrating a computer system 80. Components of computer system 80 that are not necessary for the understanding of the present invention are not shown for convenience. Computer system 80 includes a 3-D graphics system 112 coupled to a host central processing unit (CPU) 102 and a system bus 104. System memory 106 may also be coupled to system bus 104.

Host CPU 102 may be realized by any of a variety of processor technologies. For example, host CPU 102 may comprise one or more or a combination of one or more general purpose microprocessors, parallel processors, vector processors, digital signal processors, and the like. System memory 106 may include one or more memory subsystems representing other types of memory technologies. For example, system memory 106 may include random access memory (ROM), RAM (such as static RAM "SRAM", synchronous dynamic RAM "SDRAM", Rambus dynamic RAM "RDRAM" RAM, and mass storage devices. It may include.

System bus 104 may include one or more communication buses or host computer buses (for communication between the host processor and the memory subsystem). Moreover, various peripherals and peripheral buses may be connected to the system bus 104.

Graphics system 112 may be configured in accordance with the principles of the present invention and may be coupled to system bus 104 by a crossbar switch or other type of bus connection logic. The graphics system 112 drives each of the projection devices PD ₁ -PD _L and the display device 84 with corresponding video signals.

The 3-D graphics system 112 may be coupled to one or more various types of buses in addition to the system bus 104. Moreover, the 3D graphics system 112 may be coupled to a communication port, whereby it may receive graphic data directly from an external source, such as the Internet or a local area network (LAN).

The host CPU 102 may transmit / receive information to / from the graphics system 112 in accordance with programmed input / output (I / O) over the system bus 104. Alternatively, graphics system 112 may access system memory 106 by direct memory access (DMA) or intelligent bus-mastering.

A graphics application program suitable for an application programming interface (API), such as OpenGL (registered trademark of Silicon Graphics, Inc.) or Java3D (registered trademark of Sun Microsystems, Inc.), can be executed on the host CPU 102 and is a projection device. PD _1- Generates commands and data that define geometric primitives such as polygons to be output to PD _L. Host CPU 102 may then transfer the graphics data to graphics system 112 via system bus 104. In another embodiment, graphics system 112 may read the geometry data array from system memory 106 using DMA access cycles. In another embodiment, graphics system 112 may be coupled to system memory 106 via a direct port such as AGP published by Intel Corporation.

The graphics system 112 may be from various sources or from external sources such as host CPU 102 and / or other system memory 106, other memory, or a network (e.g., the Internet, etc.) Graphic data can be received from any of the sources.

As described below, the graphics system 112 may be configured to allow for more efficient microcode control, which handles subsequent color values corresponding to polygons generated by the host CPU 102. To improve performance. While graphics system 112 is depicted as part of computer system 80, graphics system 112 may also be configured as a standalone device (eg, a self-installed display). The graphics system 112 may be configured as a single chip device or as part of a chip or multi-chip module on the system.

The graphics system 112 is one of a variety of systems including network PCs, Internet devices, televisions (including HDTV systems or interactive television systems), personal digital assistants (PDAs), or other devices displaying 2D and / or 3D graphics. It can be included in anything.

As will be described further below, the 3-D graphics system of the computer system 80 may be programmed to improve the quality and realism of the image represented on the projection device PD ₁ -PD _L or the display device 84. And a sample buffer that is super-sampled with the sample-pixel computation unit. Each sample-pixel computing unit may include other hardware or filters or convolve pipelines for generating pixels in response to samples in the sample buffer. Each sample-pixel calculation unit may be operable to obtain a sample from the sample buffer and produce a pixel that is provided directly to the projection device PD ₁ -PD _L or display. The sample-pixel computing unit can operate in a "real-time" or "on-the-fly" manner.

As used herein, the terms "filter" and "convolve" may be used interchangeably. As used herein, the term “real-time” refers to a function performed at or near the refresh rate of the projection device PD ₁ -PD _L or display device. "On-the-fly" means near or at the speed required for the expressive action (i.e., motion-mix) to appear smooth and the intensity of light (i.e., flash-mix) to be seen continuously. The process or operation of generating an image at the minimum speed. These concepts are described in more detail in Russel L. De Valois and Karen K. De Valois, “Spatial Vision”, Oxford University Press, 1988.

Graphic system-Fig 3B

3 shows a block diagram illustrating an example of a graphics system 112 in accordance with the present invention. Graphics system 112 may include one or more graphics processors (GPUs) 90, one or more super-sampled sample buffers 162, and one or more sample-pixel computing units 170-1-170 -V. . Graphics system 112 may also include one or more digital-to-analog converters (DACs) 178-1 to 178-L. The graphics processing unit 90 may be a specialized graphics processor or computing unit, a multimedia processor, a DSP, or a general purpose processor.

In one example, graphics processor 90 may include one or more rendering units 150A-D. Graphics processor 90 also includes one or more control units 140, one or more data memories 152A-D, and one or more schedule units 154. Sample buffer 162 may also include one or more sample memories 160A-160N.

A. Control Unit 140

The control unit 140 acts as an interface between the graphics system 112 and the computer system 80 by controlling the data transfer between the graphics system 112 and the computer system 80. In an embodiment of the graphics system 112 that includes two or more rendering units 150A-D, the control unit 140 also directs the data flow received from the computer system 80 to the individual rendering units 150A-D. It can also be divided into the corresponding number of parallel flows sent. Graphic data may be received from computer system 80 in compressed form. Graphics data compression may advantageously reduce the transmission bandwidth required between computer system 80 and graphics system 112. In one example, control unit 140 can be configured to split the received data stream and send it to rendering units 150A-D in compressed form.

The graphics data may include one or more graphics primitives. As used herein, the terms graphics primitives refer to polygons, parametric surfaces, splines, and non-uniform rational B-splines (NURBS). ), Sub-divisions surfaces, fractals (volume primitives), volume primitives, and particle systems. This graphic prototype is described in detail in the textbook by James D. Foley, "Computer Graphics: Principles and Practice" et al., Published by Addison Wesley Publishing Co., Inc., 1996.

Embodiments and examples of the invention shown herein are described in terms of polygons for the sake of simplicity. However, any type of graphical circle may be used in place of or in addition to the polygons of these embodiments and examples.

B. Rendering Unit

The rendering units 150A-D (also referred to as draw units) here are configured to perform many functions depending on the correct performance after receiving graphics commands and data from the control unit 140. For example, the rendering unit 150A-D can decompress, transform, clip, write, and display various graphic primitives (if the received graphic data is compressed) that occur within the graphic data. to perform lighting, texturing, depth cueing, transparency processing, set-up, visible object determination, and virtual screen space rendering. Can be configured.

Depending on the type of compressed graphical data that is received, the rendering units 150A-D may perform arithmetic decoding, run-length decoding, Huffman decoding, and (eg, LZ77, And may perform dictionary decoding (such as LZSS, LZ78, LZW). In yet another embodiment, the rendering units 150A-D may be configured to decode the compressed graphic data using geometric compression. Geometric compression of 3D graphics data can significantly reduce the data size while maintaining almost the same image quality. There are two ways to compress and decompress 3D geometry:

US Patent No. 5,793,371, Application No. 08 / 511,294, filed August 4, 1994, entitled “Method And Apparatus For Geometric Compression Of Three-Dimensional Graphics Data,” Representative Registration No. 5181-05900; And

US patent application Ser. No. 09 / 095,777, filed Jan. 11, 1998, entitled “Compression of Three-Dimensional Geometry Data Representing a Regularly Tiled Surface Portion of a Graphical Object,” Agent Registration No. 5181-06602.

In embodiments of graphics system 112 that support decompression, the graphic data received from each rendering unit 150 may be decompressed into one or more graphics "primitives" and then rendered. The term primitive refers to a component of an object that defines the shape of the object (eg, a vertex, line, triangle, two- or three-dimensional polygon, polyhedron, or three-dimensional free-form surface). The rendering unit 150 may be any suitable type of high performance processor (eg, specialized graphics processor or computing unit, multimedia processor, DSPs, or general purpose processor).

Transformation refers to the application of geometric behavior to an object that contains a circle or a set of primitives. For example, objects represented in individual coordinate systems can include arbitrary positions, orientations, and sizes in real world space using the proper order of translation, rotation, scaling, and the like. The transformation can also include reflection, skewing, or other intimate transformation. More generally, the transformation may include non-linear action.

Lighting refers to calculating the illuminance of an object. Lighting calculations assign color and / or brightness to an object or selected points (eg, vertices) on the object. Depending on the shading algorithm used (e.g., Constant, Gourand, or Phong), the lighting value can be obtained at many different locations. For example, if constant shading is used (i.e. a polygonal lighted surface is assigned a constant roughness value), the lighting needs to be calculated once per polygon. When Gourand shading is used, lighting is calculated once per vertex. Phong shading calculates lighting on a per-pixel basis.

Clipping refers to the removal of a graphic primitive or a portion of a graphic prototype that lies outside of the 3-D viewing volume in real world space. The 3-D viewing volume may represent a portion of the real world space seen by the virtual observer in real world space. For example, the viewing volume can be a solid cone created by a 2-D view window and viewpoint (viewpoint location) in real world space. A solid cone can be imagined as a collection of all rays that come out of the point of view and pass through the view window. The view point may represent the real world spatial arrangement of the virtual observer. Circles or portions of circles that lie outside of the 3-D viewing volume are not currently visible and can be removed from further processing. The graphic circle or part of the graphic circle that lies within the 3-D viewing volume is the candidate to be projected into the 2-D view window.

To simplify clipping and projection calculations, the prototype can be transformed into a second, more convenient coordinate system, referred to herein as the viewport coordinate system. In viewport coordinates, the viewing volume is mapped to a canonical 3-D viewport, which may be more convenient for clipping. The term set-up refers to mapping such graphical primitives to viewport coordinates.

Graphic primitives or portions of the primitives that survive the clipping calculation may be projected into the 2-D viewport, depending on the result of the visibility judgment. Instead of clipping in 3-D, the graphic primitives can be projected onto the 2-D view plane (including the 2-D viewport) and then creep for the 2-D viewport.

Virtual display rendering refers to an operation performed to generate a sample for the projected graphic prototype. For example, the vertices of the triangle in 3-D can be projected onto the 2-D viewport. The projected triangle may comprise samples and a value (eg, red, green, blue and z values, etc.) may be assigned to a sample based on a corresponding value already determined for the projected vertices. For example, the red value for each sample in the projected triangle can be interpolated for the known red value of the vertices. These sample values for the projected triangle can be stored in the sample buffer 162. According to an embodiment, the sample buffer 16 also stores a z value for each sample. This z-value is stored with the sample for many reasons, including depth-buffering. As samples for successive circles are rendered, virtual images accumulate in the sample buffer 162. Thus, the 2-D viewport is the virtual screen on which the virtual image is rendered. The sample value including the virtual image is stored in the sample buffer 162. The points in the 2-D viewport are represented in terms of virtual screen coordinates X and Y and reside in virtual screen space.

If the virtual image is complete, for example, if all graphic primitives have been rendered, the sample-pixel computing unit 170 may access a sample containing the virtual image and filter the sample to generate pixel values. In other words, the sample-pixel operation unit 170 performs spatial convolution of the virtual image on the convolution kernel f (X, Y) to generate pixel values. For example, the red value R _p for pixel P can be calculated at place X _p , Y _p in the virtual screen space based on the following relationship:

,

Wherein the sum is calculated at samples X _k , Y _k in the neighborhood of place X _p , Y _p . Since the convolution kernel f (X, Y) is not zero only near the origin, the kernel f (X-X _p , Y-Y _p ) is not zero only near the location (X _p , Y _p ). The E value is a normalization value that can be calculated according to the following relationship.

,

Wherein the sum is calculated in the same neighborhood as above. The sum E of the normalization value may be performed in parallel with the sum for the red pixel value R _p . Place (X _p , Y _p ) is called pixel center or pixel origin. The term pixel center is used when the convolution kernel f (X, Y) is symmetric about the origin (0,0).

The pixel value can be represented by a projection device PD ₁ -PD _L for representation on the projection screen SCR. Each projection device produces an aggregated image IMG. The sample-pixel calculation unit 170 may also generate pixel values for presentation to the display device 84.

In the embodiment of the graphics system 112 shown in FIG. 3, the rendering units 150A-D calculate sample values instead of pixel values. This causes the rendering unit 150A-D to recall super-sampling, ie one or more samples per pixel. The meaning of super-sampling in the present invention is discussed in more detail below. More details about super-sampling are presented in the following book: "Principles of Digital Image Synthesis" by Andrew Glassner, 1995, Morgan Kaufman Publishing (Volume 1) and "Renderman Companion:" by steve Upstill, 1990, Addison Wesley Publishing .

The sample buffer 162 is a double-buffer such that the rendering units 150A-D can write a sample for the first virtual image into the first portion of the sample buffer 162, with the second virtual image being simultaneously Read by the sample-pixel operation unit 170 from the second portion of the sample buffer 162.

The virtual image and 2-D viewport rendered as a sample into the sample buffer 162 may correspond to an area larger than the area physically represented by the aggregated image IMG or the display image DIM. For example, a 2-D viewport can include a visible subwindow. The visible subwindow may correspond to the aggregated image IMG and / or display image DIM, while the edges of the 2-D viewport (outside the visible subwindow) allow for various effects such as panning and zooming. can do. In other words, only the portion of the virtual image lying inside the visible subwindow is physically displayed. In one example, the visible subwindow is identical to the entire 2-D viewport. In this case, all of the virtual images are physically displayed.

Note that the rendering units 150A-D may include a number of smaller and more specialized units of functionality, such as, for example, one or more set-up / decompression units and one or more lighting units.

C. Data Memories

Each rendering unit 150A-D may be combined with an instruction and data memory 152A-D. In one example, each data memory 152A-D may be configured to store data and instructions for the rendering unit 150A-D. While performance may vary, for example, each data memory 152A-D may include two 8 megabytes of SDRAM, providing a total of 16 megabytes of storage for each rendering unit 150A-D. have. As another example, RDRAM (Rambus DRAM) may be used to support decompression and set-up operations of each rendering unit, where SDRAM may be used to support the drawer function of the rendering unit. Data memory 152A-D may be referred to as "texture and render memory 152A-D."

D. Schedule Unit

The schedule unit 154 may be coupled between the rendering units 150A-D and the sampling memories 160A-N. The schedule unit 154 is configured to list the completed samples and store them in the sample memories 160A-N. Note that in larger configurations, multiple schedule units 154 can be used in parallel. In one example, the scheduling unit 154 may be performed as a crossbar switch.

E. Sample Memories

The super-sampled sample buffer 162 includes sample memories 160A- 160N, which are configured to store a plurality of samples generated by the rendering unit. As used herein, "sample buffer" refers to one or more memories that store samples. As mentioned above, one or more samples are filtered to form an output pixel. The output pixel value can be provided to the projection device PD ₁ -PD _L for display on the projection screen SCR. Axial pixel values may also be provided to the display device 84. Sample buffer 162 may also be configured to support super-sampling, critical sampling, or sub-sampling depending on the pixel resolution. In other words, the average distance between samples X _k , T _k in the virtual image (stored in sample buffer 162) may be less than, equal to, or greater than the average distance between pixel centers in the virtual screen space. Moreover, since the convolution kernel f (X, Y) can take a non-zero functional value around the multiple pixel centers, a single sample can contribute to multiple output pixel values.

Sample memories 160A-160N may include any of a number of different types of memory (eg, SDRAMs, SRAMs, RDRAMs, 3DRAMs, or next generation 3DRAMs) in various sizes. In one example, each schedule unit 154 is coupled with four sample memory banks, each bank comprising four 3DRAM-64 memories. At the same time, the 3DRAM-64 memory can form a 116-bit deep super-sampled sample buffer that stores multiple samples per pixel. For example, in one example, each sample memory 160A-160N can store 16 samples per pixel.

3DRAM-64 memory is a specialized memory with a single chip Z of one chip and configured to support full internal double buffering. The double buffer portion contains two RGBX buffers, where X is the fourth channel used to store other information (eg alpha). The 3DRAM-64 memory also has a lookup table that controls an internal 2-1 or 3-1 multiplexer that accepts window ID information and selects which buffer's contents are to be output. 3DRAM-64 memory is the next generation of 3DRAM memory that will soon be available by the Semiconductor Group of Mitsubishi Electric Corporation. In one example, the four chips used in combination are sufficient to create a 1280 X 1024 super-sampled sample buffer with a double buffer. Since 3DRAM-64 memories are internally double-buffered, the input pins for each of the two frame buffers in a double-buffered system are multiplexed over time (using a multiplexer in memory). Output pins can likewise be multiplexed by time. This reduces the pin count while providing the benefit of a double buffer. 3DRAM-64 memory further reduces pin count by not having a z output pin. Since the z comparison and memory buffer selection are handled internally, this can simplify the sample buffer 162 (eg, with little or no selection logic on the output side). Using 3DRAM-64 also reduces memory bandwidth because information can be read into memory without the prior art process of reading data, performing z comparisons, and then rewriting the data. Instead, data can simply be written to 3DRAM-64 as the memory performs the steps described above internally.

However, in other embodiments of graphics system 112, other memories (eg, SDRAMs, SRAMs, RDRAMs, or current generation 3DRAMs) may be used to form sample buffer 162.

Graphics processor 90 may be configured to generate a plurality of sample positions in accordance with a particular sample position diagram (eg, regular grid, disturbed normal grid, etc.). Alternatively, the sample positions (or offsets added to the regular grid position to form the sample position) can be read from the sample position memory (eg, RAM / ROM table). Upon receiving the polygon to be rendered, the graphics processor 90 determines which sample falls within the polygon based on the sample location. The graphics processor 90 renders a sample corresponding to the polygon and stores the rendered sample in the sample memories 160A-N. As used herein, note that the terms "render" and "draw" can be used interchangeably and refer to calculating the color value for a sample. Depth values, alpha values and other values for the sample may also be calculated in the rendering or drawer process.

F. Sample-to-pixel Calculation Units

The sample-pixel operation units 170-1-170-V (hereinafter collectively referred to as sample-pixel operation units 170) are placed between the sample memory 160A-N and the DACs 178-1 through 178-L. Can be combined. The sample-pixel operation unit 170 reads the selected sample from the sample memory 160A-N and performs convolution (ie, a filter action) on the sample to output the output pixels to the DACs 178-1 to 178-L. Configured to generate a value. The sample-pixel operation unit 170 may be programmed to perform different filter functions at different times, depending on the type of output desired. In one example, the sample-pixel operation unit 170 performs a 5x5 super-sample reconstruction band-pass filter to convert the data of the super-sampled sample buffer (stored in the sample memories 160A-N) into a single pixel value. Convert. In another embodiment, the computing units 170A-D filter the selected number of samples to calculate the output pixels. The selected sample may be multiplied by a spatial weighting factor that weights the sample based on its position along the center of the pixel being calculated. The filtration action may be any of a variety of filters and may be used alone or in combination. For example, tent filters, round and elliptical filters, Mitchell filters, band pass filters, sink function filters, and the like may be used for the convolutional action.

Sample-pixel computing unit 170 may also be formed with one or more of the following features: color search using a pseudo color table, direct color, inverse gamma correction, filtering of samples into pixels, programmable gamma encoding, and selection. Color space conversion. Other features of the sample-pixel operation unit 170 include a programmable video timing generator, a programmable pixel clock synthesizer, an edge-mix function, a hotspot correction function, a color space and a crossbar function. Once the sample-pixel computation unit adjusts the timing and color of each pixel, the pixels are output to the DACs 178-1 to 178-L.

G. DACs

Digital-to-analog converters (DACs) 178-1 to 178 -L (hereinafter collectively referred to as DACs 178) operate as the final output stage of graphics system 112. The DACs 178 convert the digital pixel data received from the operation unit 170 into an analog video signal. Each of the DACs 178-1 through 178-L may be coupled to a corresponding one of the projection devices PD ₁ -PD _L. The DAC 178-1 receives the digital pixel data of the first stream from the one or more computing units 170 and converts the first stream into a first video signal. The first video signal is provided to the projection device PD ₁ . Similarly, the DACs 178-1 through 178-L receive corresponding streams of digital pixel data, which are converted into corresponding analog video signals and corresponding to among the projection devices PD ₁ -PD _L. Provided in one

As an example, note that the DACs 178 may be ignored or completely omitted to output as digital pixel data instead of analog video signals. This is useful when the projection device PD ₁ -PD _L is based on digital technology (eg LCD-type display or digital micro-mirror display).

Super-sampling-Figure 4-5

4 shows a portion of the virtual screen space for the non-super sampled example. Dots represent sample locations, and rectangular boxes superimposed on virtual screen space define pixel boundaries. One sample is placed in the center of each pixel, and red, green, blue, z and the like are calculated for that sample. For example, sample 74 is assigned to the center of pixel 70. Although the rendering unit 150 only calculates one sample per pixel, the sample-pixel calculation unit 170 can still calculate the output pixel value based on multiple samples, for example by using a convolution filter that spans several pixels. .

Referring now to FIG. 5A, an example of super-sampling is shown. In this embodiment, two samples are calculated per sample. Samples are distributed along a regular grid. Even if there are more samples than pixels in the figure, for example, by discarding all samples except the sample closest to the center of each pixel, the output pixel value can be calculated using one sample per pixel. However, there are many advantages in computing pixel values based on multiple samples.

A support region 72 is superimposed on the pixel 70, indicating support of a filter disposed in the pixel 70. Filter support is a set of places where a filter takes a non-zero value (that is, a filter kernel). In this example, the support area 72 is shaped like a circular disk. The output pixel values (eg, red, green, blue and z values) for pixel 70 are determined only by samples 74A and 74B. This is because these are the only samples corresponding to the inside of the support area 72. This filtering action can advantageously enhance the realism of the image being represented, by smoothing the steep edges of the image to be represented (ie, anti-aliasing). Filtering simply averages the samples 74A-B to form the final value of the output pixel 70, or increases the contribution of the sample 74B (at the center of the pixel 70) and increases the sample 74A ( That is, the contribution of the sample far away from the center of the pixel 70 can be reduced. The filter 72, i.e., the support region 72, is rearranged for each output pixel being computed so that the center of the support region 72 coincides with the center position of the pixel being computed. Other filters and filter location diagrams are also possible and can be contemplated.

Referring now to FIG. 5B, another embodiment of super-sampling is shown. In this example the samples are randomly placed. Thus, the number of samples used to compute the output pixel may vary from pixel to pixel.

Super-sampled sample buffer and real-time sample-pixel operation-Figure 6-13

6 illustrates one possible configuration for the flow of data, through an example of graphics system 112. As shown in the figure, geometric data 350 is received by graphics system 112 and used to conduct drawer process 352. The drawer process 352 is performed by one or more control units 140, rendering units 150, memory 152, and scheduling units 154. Geometric data 350 includes data for one or more polygons. Each polygon contains a plurality of vertices (eg three vertices in the case of a triangle), some of which may be shared. Data such as x, y and z coordinates, color data, lighting data and texture map information may be included for each vertex.

In addition to the vertex data, the drawer process 352 (which may be done by the rendering units 150A-D) also receives sample coordinates from the sample location memory 354. The sample location information defines the location of the sample in virtual screen space, i.e. the 2-D viewport. The drawer process 352 selects a sample corresponding to the polygon currently being rendered, and each of these locations according to each position within the polygon. Assemble a set of values for a sample (eg, depth of red, green, blue, z, alpha, and / or field information). For a sample, the z value of the sample that falls inside the triangle can be interpolated from the known z values of the three vertices. Each set of calculated sample values is stored in a sample buffer 162.

In one example, the sample location memory 354 can be embodied inside the rendering units 150A-D. As another example, the sample location memory 354 may be recognized as part of the memory 152A-152D or as a separate memory.

The sample location memory 354 can store the sample locations in terms of their virtual screen coordinates (X, Y). Alternatively, sample location memory 354 may be configured to store only offsets dX and dY for the sample relative to the location on the regular grid. Storing only the offset may use less storage space than storing the global coordinates (X, Y) for each sample. The sample position information stored in the sample position memory 354 can be read by a given sample position calculating unit (not shown) and processed to calculate an example sample position for the graphics processor 90. More detailed information on the sample position offset is included below (see the description of FIGS. 9 and 10).

In another embodiment, sample location memory 354 may be configured to store a table of random numbers. Sample location memory 354 may also include given hardware to create one or more other types of regular grids. This hardware can be programmed. The stored random number will be added as an offset towards the regular grid position generated by the hardware. In one example, the sample location memory can be programmed in many different ways to access or expand the random number table, thus providing clearer randomness for a random number table of a given length. Thus, fewer tables can be used without generating the starting artifacts caused by simple repetition of the sample position offset.

The sample-pixel calculation unit 360 uses the same sample position as in the drawer process 352. Thus, in one example, the sample location memory 354 may generate a series of random offsets to calculate the sample location for the process 352, after which the same for the sample-pixel calculation process 360 The same order of random offsets for calculating the sample position can be regenerated.

As shown in the figure, the sample location memory 354 can be configured to store sample offsets generated according to a number of other designs, such as normal square grids, regular hexagonal grids, disturbed normal grids, or random (probabilistic) distributions. have. Graphics system 112 may receive instructions from geometry data 350 indicating which operating system, device driver, or type of sample location map to use. Thus, sample location memory 354 may be configured or programmed to generate different location information in one or more other designs. More detailed information regarding the various sample location illustrations will be described further below (see the description of FIG. 8).

In one example, the sample location memory 354 may include a RAM / ROM that stochically contains the determined sample point or sample offset. Thus, the density of samples in the virtual screen space may not be uniform if observed at a small scale. Two bins of the same area centered at different places in the virtual screen space may have different numbers of samples. As used herein, the term “bin” refers to an area or area within a virtual screen space.

The array of bins can overlap on the virtual screen space. That is, the storage of samples in the 2-D viewport and sample buffer 162 may be organized in terms of bins. The sample buffer 162 includes an array of memory blocks corresponding to bins. Each memory block may store sample values (eg, red, green, blue, z, alpha, etc.) for corresponding samples in a corresponding bin. The memory block can easily be computed from the corresponding empty location in the virtual screen space, and vice versa with an address that can know the location from the calculation. Thus, use of the bin can simplify the storage and access of sample values in the sample buffer 162.

Bins can be attached to 2-D viewports in regular arrays such as square arrays, rectangular arrays, triangular arrays, hexagonal arrays, or irregular arrays. The size and shape can be programmed. The maximum number of samples that can exist in a bin is determined by the storage space allocated to the corresponding memory block. The maximum number of samples is referred to herein as the empty sample capacity or simply called the bin capacity. The empty capacity can take any of a variety of values. The free capacity value can be programmed. Here, the memory block in the sample buffer 162 corresponding to the bin in the virtual screen space will be referred to herein as a memory bin.

The specific location of each sample in one bin can be determined by searching for the offset of the sample in the RAM / ROM table, i. However, depending on the implementation, not all choices of free capacity have a unique set of offsets stored in the RAM / ROM table. The offset for the first free capacity value may be determined by accessing an offset of one subset stored for the second larger capacity value. In one example, each bin capacity value supports at least four different sample location plots. Using different sample location diagrams can reduce the final image artifacts due to repeated sample locations.

In one example, sample location memory 354 may store several pairs of 8-bit numbers, each pair comprising an x-offset and a y-offset. (Other offsets are also possible, such as time offsets, z-offsets, etc.). When added to an empty position, each pair defines a particular position in the virtual screen space, a 2-D viewport. To improve read access time, the sample location memory 354 can be built in a wide / parallel manner so that the memory can output one or more sample locations per read cycle.

Once the sample location is read from the sample location memory 354, the draw process 352 selects a sample corresponding to the polygon currently being rendered. The drawer process 352 then calculates z and color information (which may include alpha or other depth of field information) for each respective sample and stores the data in the sample buffer 162. In one example, the sample buffer may only single-buffer the z value (and alpha value) while double buffering other sample components, such as color. Unlike conventional systems, graphics system 112 may double-buffer all samples (although not all sample components may be double-buffered, ie, samples may have components that are not double-buffered). have. In one example, the samples are stored in a bin of sample buffer 162. In some embodiments, the capacity of the bin may vary from frame to frame. Moreover, the bin capacity may vary spatially for each bin of a single frame rendered into the sample buffer 162. For example, a bin at the edge of the 2-D viewport may have a smaller specific capacity than the bin corresponding to the center of the 2-D viewport. Since observers pay most of their attention to the center of the screen SCR or display image DIM, more processing bandwidth can be provided to provide improved image quality at the center of the 2-D viewport. The size and appearance of the bins may also vary from region to region or from frame to frame. The use of the bean will be described in more detail below.

In parallel and independent of the drawer process 352, the filtration process 360 reads (a) the sample position from the sample position memory 354, (b) the corresponding sample value from the sample buffer 162, and (c) Filter the sample value and (d) output the resulting output pixel value to one or more projection devices PD ₁ -PD _L and / or display device 84. The sample-pixel calculation unit 170 performs the filtration process 360. The filtration process 360 may be operable to generate red, green, and blue values for the output pixel based on spatial filtration of the corresponding data for the selected plurality of samples, i.e., samples corresponding to the perimeter of the pixel center. Other values such as alpha can also be generated. In one example, filtration process 360 includes: (i) determining the distance from each sample to the center of the output pixel being filtered; (ii) multiply the characteristic value (eg red, green, blue, alpha) of each sample by the filter weight which is a specific (programmable) function of the distance of the sample; (iii) one sum per characteristic (e.g., sum of red, sum of green, etc.), to generate the sum of the weighted characteristic values and (iv) normalize the sum to produce corresponding pixel characteristic values. . Filtration process 360 is described in more detail below (see the description with respect to FIGS. 11, 12 and 14).

In the described embodiment, the filter kernel is a function of the distance from the pixel center and thus radially symmetrically. However, in other embodiments, the filter kernel is a more general function of the displacement of X and Y from the pixel center. Thus, the support of the filter, i.e. near 2-D where the filter kernel takes a non-zero value, may not be a circular disk. Any sample that falls within the support of the filter kernel can affect the output pixel being computed.

Referring to FIG. 7, a diagram illustrating another embodiment of the graphics system 112 is shown. In this embodiment, two or more sample position memories 354A and 354B are utilized. Therefore, the sample position memory 354A-B must be a double-buffer. If the sample position remains the same from frame to frame, the sample position may be a single buffer. However, if the sample position can vary from frame to frame, the graphics system 112 can be advantageously configured to double-buffer the sample position. The sample location may be a double buffer on the rendering side (ie, memory 354A may be a double buffer) and / or a double buffer on the filter side (eg, memory 354B may be a double buffer). Can be Other combinations are also possible. For example, memory 354A may be a single-buffer while memory 354B may be a double buffer. This configuration allows the other side of memory 354B to be used for filtration process 360 while one memory 354B is used for refresh. In this configuration, the graphics system 112 can change the sample position diagram per frame by moving the sample position (or offset) from the memory 354A to the double-buffered memory 354B as each frame is rendered. . Thus, the sample locations stored in memory 354A and used by the drawer process 352 to render the sample values may be duplicated into the memory 354B for use by the filtering process 360. Once the location information has been duplicated into memory 354B, location memory 354A may carry the new sample location (or offset) used for the second frame to be rendered. In this way, sample location information follows the sample value from drawer process 352 to filtration process 360.

However, in another embodiment, a tag may be stored at an offset for the samples in the sample buffer 162 being super-sampled. This tag can be used to retrieve the offset (side, disturb) dX and dY for each particular sample.

Sample location diagram

8 shows many different sample location diagrams. In the normal grid position diagram 190, the samples are placed in a fixed position relative to the normal grid superimposed on the 2-D viewport. For example, the samples may be located in the center of a square drawn by a regular grid. More generally, the tiling of any 2-D viewport can produce a canonical position plot. For example, the 2-D viewport can be tiled in triangles, so that the sample can be located at the center (or corner points) of the triangle tile. Semi-normal tiling such as hexagonal tiling, algebraic tiling, and Penrose tiling are also contemplated.

In the disturbed normal grid position diagram 192, the sample positions are defined in terms of disturbances from a set of fixed positions on the normal grease or tiling. In one example, samples may be moved from their corresponding fixed grid position by a random x and y offset or by a random angle (from 0 degrees to 360 degrees) and a random radius (from 0 to maximum radius). The offset can be generated in a number of ways, for example, by using a pseudo-random function, by reading a table of stored offsets, by hard on a small number of seeds. In other words, the disturbed normal grid diagram 192 can be based on any type of normal grid or tiling. Samples are produced by disturbances relative to the grid (eg hexagonal tiling is particularly preferred because of the geometrical nature of this configuration).

Probabilistic sample location plot 194 represents a third potential design type for placing the sample. Probabilistic sample locations include randomly distributing samples across the 2-D viewport. The random location of the sample can be achieved by a number of different methods, for example by using a random number generator such as an internal clock to generate a pseudo random number. The random number or location may also be precomputed and stored in memory. Note that, as used in this application, random locations can be selected from statistical populations (eg, Poisson-disk distribution. Other types of random and pseudo-random locations are known as "Principles of Digital Image Synthesis ”, by Andrew S. Glassner, Morgan Kaufman Publishers, 1995, Chapter 10 of the first volume of the paper.

Referring now to FIG. 9, an example of a disturbed regular grid design 192 is shown. In this embodiment, the samples have randomly offset by x- and y- offsets from the regular square grid. As the magnified area indicates, the sample 198 has an x-offset 134 that characterizes the horizontal displacement from the corresponding grid intersection 196. Random x-offset 134 and y-offset 136 may be limited to particular range values. For example, when X _max is the width of the grid rectangle, the x-offset may be limited to the range of 0 to X _max . Similarly, at the height of the Y _max grid rectangle, the y-offset can be limited to the range from 0 to Y _max . Random offsets may also be specified by angle and radius with respect to grid intersection 196.

10 illustrates another embodiment of a disturbed regular grid design 192. In this embodiment, the samples are grouped into "bins" 138A-D. In this embodiment, each bin contains nine samples, i.e., has a bin capacity of nine. In other embodiments, other bins may be used (eg, bins storing four samples, sixteen samples). The location of each sample is determined as an offset relative to the origin of the bin in which it resides. The origin of the bin may be selected as the lower left corner of the bin (or any other convenient place inside the bin). For example, the location of sample 198 may be x-offset 124 and y-offset 126. ) Is added to the x and y coordinates of the origin 132D of the bin 138D, respectively. As described above, this may reduce the size of the sample position memory used in some embodiments.

Figure 11-Convert Samples to Pixels

As discussed above, the 2-D viewport can be covered with an array of spatial bins. Each spatial bin may be filled with a sample whose location is determined by sample location memory 354. Each spatial bin corresponds to a memory bin in sample buffer 162. The memory bin stores sample values (red, green, blue, z, alpha, etc.) for the samples present in the corresponding spatial bins. The sample-pixel calculation unit 170 (also referred to as the convex unit 170) is configured to read the memory bin from the sample buffer 162 and convert sample values contained in the memory bin into pixel values.

Parallel Sample-Pixel Filtration with Columns-Figures 11A-11B

11A illustrates one possible method for quickly converting a sample stored in sample buffer 162 into pixels. The spatial bins covering the 2-D viewport can be organized into columns (eg, columns 1-4). Each column contains a two-dimensional sub-array of spatial bins. The columns can be configured to overlap horizontally (eg, by one or more bins). Each sample-pixel operation unit 170-1 through 170-4 may be configured to access a memory bin corresponding to one of the columns. For example, the sample-pixel operation unit 170-1 may be configured to access a memory bin corresponding to the spatial bin of column 1. The data path between the sample buffer 162 and the sample-pixel operation unit 170 can be optimized to support this column-based communication.

The amount of overlap may depend on the horizontal diameter of the filter support for the filter kernel used. The example shown in FIG. 11A shows an overlap of two bins (such as square 188) representing a single bin containing one or more samples. Advantageously, this configuration allows the sample-pixel operation unit 170 to work independently and in parallel, with each sample-pixel operation unit 170 receiving and converting the column. Overlapping columns can prevent visual bands or other artifacts from appearing at column boundaries for operators larger than a range of one pixel.

Moreover, the embodiment of FIG. 11A includes a plurality of bin caches 176 coupled to the sample buffer 162. In addition, each empty cache 176 is associated with a corresponding one sample-pixel operation unit 170. A typical bin cache 176-I (I takes any of a positive integer value) stores a set of memory bins corresponding to column I and acts as a cache for the sample-pixel operation unit 170-I. The generic bin cache 176-I may have an optimized coupling to the sample buffer 162 to facilitate access to the memory bins for column I. The sample-pixel operation for two adjacent output pixels may include many of the same bins, and the bin cache 176 may increase the overall access bandwidth for the sample buffer 162. The sample-pixel operation unit 170 can be performed in many different ways, including a high performance arithmetic logic unit (ALU) core, a functional unit from a microprocessor or DSP, or custom design using hardware multipliers and adders, and the like. .

Referring now to FIG. 11B, another method of performing parallel sample-pixel operations is shown. In this embodiment, the sample buffer 162 is divided into a plurality of vertical columns or rows as in the example described above. However, the columns in this embodiment are not the same size or width. For example, column 1 contains significantly fewer sample bins than column 4. This embodiment is particularly useful for the construction of graphics systems that support variable sample densities. As mentioned above and described in more detail below, the graphics system is a sample buffer that corresponds to the region of the final image that is most beneficial at higher sample densities, i.e., the image region corresponding to the viewer's retina and the viewer's particular region of interest. May devote more samples (ie, higher sample density) for the region of 162. In this system that supports variable sample densities, the ability to vary the width of the column allows the graphics system to advantageously equalize the number of samples filtered by each sample-pixel computing unit. For example, column 1 may correspond to the portion of the image that is represented where the center of the viewer's point of view is concentrated. Thus, the graphics system can provide high density samples in column 1 of the bin and the graphics system can provide low density samples in column 4 of the bin. Thus, by decreasing the width of column 1 and increasing the width of column 4, the sample-pixel computing units 170-1 and 170-4 can each filter approximately the same number of samples. Advantageously, balancing the filtration burden between the sample-pixel computing units allows the graphics system to use the processing resources in a more efficient manner.

In some embodiments, the graphics system may be configured to fluidly change the width of the column on a frame-by-frame basis (or on a portion of a frame). In embodiments of a graphics system that fluidly varies the sample density (eg, eye-tracking, retina and dot tracking, hero tracking), the sample density can vary on a frame-by-frame basis, thus reducing the color width on a frame-by-frame basis. Changing allows the computational resources of the sample-pixel operation unit 170 to be utilized in a more efficient manner. In some embodiments, the column width may vary on a scan line reference or at some other time-reference. In addition to changing over time, as the figure shows, the columns can also be configured to overlap (as in the embodiments described above) to prevent visual artifacts (eg, cracks, tears, or vertical lines) from appearing. have.

Parallel Sample-Pixel Filtration Using Rows-Figure 12

Referring now to FIG. 12, another embodiment of a graphics system is shown. In this embodiment, the sample buffer 162 is divided into a plurality of horizontal rows or rows. As in the previous embodiment, the rows may overlap and / or change in width to compensate for varying sample densities. As in the previous embodiment, each row may provide bins (and samples) to a particular bin cache 176 and corresponding sample-pixel computing unit 170.

Parallel Sample-Pixel Filtration Using Rows-Figure 13

Referring now to FIG. 13, another embodiment of a graphics system is shown. In this embodiment, the sample buffer 162 is divided into a plurality of rectangular regions. As in the previous embodiments, rectangular regions may overlap (eg, on a frame-by-frame or scanline basis), have different sizes, and / or may vary fluidly in size. Each region may be configured to provide bins (and samples) to a particular bin cache 176 and corresponding sample-pixel computing unit 170. In some embodiments, each rectangular area may correspond to an image projected by one of a plurality of projectors (eg, an LCD projector). In other embodiments, each rectangular area may correspond to a particular portion of a single image that is displayed or projected on a single display device. As with the previous embodiment, advantageously, the sample-pixel computing unit 170 can be configured to operate independently, in parallel, to reduce latency of the graphics system. As mentioned earlier, the rectangular areas shown in FIG. 13 need not be of uniform size and / or shape.

In an example of a graphics system having a varying area size or row width, the amount of overlap can vary fluidly on a frame-by-frame or sub-frame basis. Other shaped areas in which the sample buffer 162 may be divided are possible and contemplated. For example, in some embodiments each sample-pixel computing unit may receive bins (and samples) from a plurality of small regions or rows.

In some embodiments, sample cache 176 may have sufficient storage space to store the entire horizontal scanline. For this reason, it may be useful to divide the sample buffer into regions. Depending on the display device, the regions may be part of an odd-only and even-only scanline. In some systems, such as systems with multiple display devices, each region may correspond to a single display device or quadrant of the image being displayed. For example, assuming that the images formed by the four projectors are pasted together to form a single, large image, each sample-pixel calculation unit may receive samples corresponding to the pixels displayed by the particular projector. In some embodiments, the region of the overlapping station may be stored twice, whereby each sample-pixel operation unit has exclusive access to a particular area of the sample buffer. This can avoid timing problems that appear when two different sample-pixel computing units (or two sample cache controllers) attempt to access the same set of memory locations at the same time. In another embodiment, the sample buffer may be multi-ported to allow for more than one concurrent access to the same memory location.

As mentioned above, in some embodiments the sample cache is configured to read samples from the sample buffer. In some embodiments, the sample may be read on a bin-to-bin basis from the sample buffer. The sample cache and / or sample buffer may contain control logic that is configured such that all samples potentially contributing to one or more pixels being filtered (or about to be filtered) may be used for the corresponding sample-pixel operation unit. It may include. In some implementations, the sample cache may be large enough to store a predetermined array of bins, such as 5x5 bins (to match the maximum filter size). In another embodiment, instead of a 5x5 empty cache, the sample cache may be configured to output pixels while accumulating in a series of multiple accumulators. In this embodiment, different coefficients are generated for each pixel, depending on the number of samples and their weights.

How to read a sample from the sample buffer-Fig. 14

Referring now to FIG. 14, an example of a method of reading sample values from a super-sampled sample buffer is shown in more detail. As shown in the figure, the convolution filter kernel 400 traverses column I (in the direction of arrow 406) to produce an output pixel, where index I is any value in the range of 0 to 4. Can be taken. The sample-pixel operation unit 170 -I may perform the sample-pixel filter kernel 400. Bin cache 176-I may be used to provide quick access to the memory bin corresponding to column I. For example, because the sample-pixel filter kernel 400 covers a 5 to 5 array of spatial bins, the bin cache 176 -I may have a capacity equal to or greater than 25 memory bins. As the action of converting the sample into pixels proceeds, the memory bin is read from the sample buffer 162 being super-sampled and stored in the bin cache 176-I. In one example, bins 410 that are no longer needed, such as bins 410, are overwritten by the new beans in the bin cache 176-I. As each output pixel is generated, the sample-pixel filter kernel 400 moves. Kernel 400 may be envisioned as proceeding in the direction indicated by arrow 406 in a sequential manner within column I. When the kernel 400 reaches the end of the column on the right boundary 404 of column I, it moves to a bin below one or more rows and again begins to progress horizontally from the left column boundary 402. Thus, the sample-pixel calculation unit proceeds in a scan line fashion, producing a continuous column of output pixels for display.

15 shows potential boundary conditions in the calculation of output pixel values. 2-D viewport 420 is shown as a rectangle covered with a rectangular array of spatial bins. Recall that all spatial bins correspond to memory bins in sample buffer 162. The memory bin stores sample values and / or sample locations for samples that exist in the corresponding spatial bin. As described above, the sample-pixel calculation unit 170 filters the sample around the pixel center to produce an output pixel value (eg, red, green, blue, etc.). Pixel center PC ₀ is sufficiently close to the lower side boundary (Y = 0) of 2-D viewport 420 where filter support 400 is not entirely contained within the 2-D viewport. The sample-pixel operation unit 170 may generate sample values for the sample position and / or the edge portion of the filter support 400 (ie, the portion outside the 2-D viewport 420) by various methods. have.

In one example, the sample-pixel operation unit 170 may generate one or more dummy bins to cover the edges of the filter support 400. The sample location for the dummy bin is generated by reflecting the sample location of the spatial bin across the 2-D viewport boundary. For example, the dummy bins (F, G, H, I, J) are assigned by reflecting the sample position across the boundary line Y = 0 corresponding to each spatial bin (A, B, C, D, E). May be a sample position. The predetermined color value may be associated with a dummy sample inside the dummy bin. For example, a value (0, 0, 0) for an RGB color vector can be assigned to each dummy sample. As pixel center PC ₀ moves downwards (ie, through it towards boundary Y = 0), additional dummy bins with dummy samples are created (moving along pixel center PC ₀ ) and filter support ( 400). The number of dummy bins within the filter support 400 increases and reaches a maximum when the filter support 400 has moved entirely out of the 2-D viewport 420. Accordingly, the filter value calculated based on the filter support 400 approaches the predetermined background color as the pixel center PC ₀ crosses the boundary.

The pixel center may be placed outside of the 2-D viewport 420 and, moreover, may be close enough to the viewport boundary so that some of the filter support can be placed within the 2-D viewport 420. Filter support 401 corresponds to one such pixel center. The sample-pixel operation unit 170 may generate dummy bins Q, R, S, T, U, and V to cover the outer portion of the filter support 401 (ie, the outer portion of the 2-D viewport). have. The dummy bins Q, R, and S may be assigned sample positions based on the sample positions of the spatial bins N, O, P and / or the spatial bins K, L, and M.

The sample position for the dummy bin may also be switched on-the-fly by changing the sample position in response to the spatial bin across the viewport boundary, or according to a regular, disturbed normal or stochastic sample position plot. -fly) to generate sample positions.

16 illustrates another embodiment of a method for performing pixel value calculation. The sample-pixel calculation unit 170 may perform pixel value calculation using the visible subwindow 422 of the 2-D viewport 420. The visible subwindow is depicted as a rectangle with a lower left corner (X ₁ , Y ₁ ) and an upper right corner (X ₂ , Y ₂ ) in virtual screen space. Note that in some embodiments, the filter may be auto-normalized or pre-normalized to reduce the number of operations required to determine the final pixel value.

Rendering sample pixels into a super-sampled sample buffer-Figure 17

FIG. 17 is a flowchart of an example of a method of drawing or rendering a sample pixel into a super-sampled sample buffer. Some steps in FIG. 12 may occur simultaneously or in a different order. In step 200, the graphics system receives graphics commands and graphics data directly from the host CPU 102 or main memory 106. In step 202, the instructions and data are sent to one or more rendering units 150A-D. In step 204, the rendering unit 150A-D determines whether the graphic data is compressed. If the graphic data is compressed, the rendering units 150A-D decompress the data into an available format, for example triangle, as shown in step 206. Next, the triangle is transformed, for example, from model space to world space And the like, and the like (step 208A).

If the graphics system performs variable resolution supersampling, the triangle is compared to a set of sample density region boundaries (step 208B). In variable-resolution super-sampling, other regions of the 2-D viewport may be different sample densities assigned based on a number of factors (e.g., the center of attention of the observer's projection screen SCR as determined by eye or head tracking). have. The sample density area is described in more detail below (see the item entitled Variable Resolution Sample Buffer below). If the triangle crosses the sample-density region boundary (step 210), the triangle may be divided into two smaller polygons along the region boundary (step 212). Polygons can be subdivided into triangles again if necessary (since triangles are usually split). In one example, graphics system 112 may be configured to render the original triangle twice, ie, once for each sample density, then cut the two versions and divide them into two respective sample density regions.

In step 214, one of the sample location plots (eg, normal grid, disturbed normal grid, or stochastic) is selected from sample location memory 354. The sample position diagram is generally preprogrammed into the sample position memory 354, but may also be selected as "on the fly." In step 216, based on the selected sample location diagram and the size and appearance of the spatial bins, the rendering unit 150A-D determines which bins will contain the samples disposed within the area of the triangle. In step 218, the offsets dX and dY for the samples in these bins are read from the sample location memory 354. In step 220, the location of each sample is calculated using the coordinates of the empty origin corresponding to the offsets dX and dY, and compared with the vertices of the triangle to determine if the sample is inside the triangle. Step 220 is discussed in more detail below.

For each sample determined to be inside the triangle, the rendering unit draws the sample by calculating the color, alpha, and other characteristics of the sample. This may include interpolation and lighting operations based on texture map information and color associated with the vertices of the triangle. Once the sample is rendered, it is sent to the scheduling unit 154, which then stores the sample in the sample buffer 162 (step 224).

Embodiments of the rendering method described above are used for illustrative purposes only and are not intended to be limiting. For example, in some embodiments, the steps shown as occurring serially in FIG. 13 may be performed in parallel. Moreover, in some embodiments of the graphics system, some steps (eg, steps 204-206 in embodiments that do not perform geometric compression), or steps in embodiments that do not perform variable resolution super-sampled sample buffers. 201-212 may be reduced or deleted.

Determining which sample exists inside the polygon being rendered-Figure 18

Comparing which samples exist inside the polygon being rendered can be done in a number of different ways. In one example, first, a delta between three vertices defining a triangle is determined. For example, this delta is the first vertex from the first vertex (v2-v1) = d12, the second vertex from the third vertex (v3-v2) = d23, and the third vertex again from the third vertex (v1-v3). ) = d31. These deltas form a vector, and each vector can be classified as belonging to one of four quadrants of the coordinate plane (eg, by using two sign bits of the delta X coefficient and the delta Y coefficient). A third condition can be added that determines whether the vector is an X-major vector or a Y-major vector. This may be determined by calculating whether the abs (delta_x) value is greater than the abs (delta_y) value. By using this three bits of information, the vectors can each be classified as belonging to one of eight different regions of the coordinate plane. If 3 bits are used to define this area, then the X-sign bit (shifted left by 2), the Y-sign bit (shifted left by 1), and the X-major bit are shown in FIG. It can be used to create eight regions.

Next, three edge equations can be used to define the inner portion of the triangle. These edges can be represented by a line of either (or both) of the form y = mx + b or x = ry + c at rm = 1. To reduce the range of numbers needed to express the slope, two X-major and Y-major equation forms can be used for the edge equation (so that the absolute value of the slope is in the range of 0 to 1). Thus, the edge (or semi-plane) inequality can be expressed in one of two corresponding forms of:

X-major: y-m-x-b < 0, where point (x, y) is below the edge;

Y-major: x-r-y-c < 0, where point (x, y) is to the left of the edge;

If the point of (x, y) lies below the line defined by the edge, the X-major inequality produces a logical true value (that is, a sign bit equal to 1). If the point of (x, y) is to the left of the line defined by the edge, the Y-major equation produces a logical true value. The side including the interior of the triangle is open for each linear inequality, which can be specified by a Boolean variable called the accept bit here. Thus, the sample (x, y) is on the inner side of the edge if

X-major: (y-m.x-b <0) <xor> accept = true;

Y-major: (x-m-y-b <0) <xor> accept = true;

The accept bit for a given edge is obtained when (a) the region where the edge delta vector is present (from 0 to 7) and (b) when the crosswise clockwise is cw = 1 and the counterclockwise crossover is cw = 0. Based on the intuition of the edge crossing, it can be calculated according to the following table. The symbol "!" Is a logical complement.

1: Accept =! Cw

0: Accept = cw

4: accept = cw

5: accept = cw

7: Accept = cw

6: Accept =! Cw

2: Accept =! Cw

3: Accept =! Cw

Breaking this presentation rule is also done (e.g., the coordinate axis can be defined as belonging to a positive eighth space). Similarly, an X-major can be defined as having all points balanced on the slope.

As another example, the accept side of an edge can be determined by applying the edge inequality to the third vertex of the triangle (ie, the vertex that is neither of the two vertices forming the edge). This method can incur a large additional cost and can be avoided by the technique described above.

In order to determine the "faced-ness" of the triangle (ie to determine if the triangle is clockwise or counter-clockwise), the delta-direction of the two edges of the triangle is examined and the two edges The slope of can be compared. For example, assuming that edge 12 has a delta-direction of 1 and the second edge (edge 23) has a delta-direction of 0, 4, or 5, the triangle is counterclockwise. However, if edge 23 has a delta-direction of 3, 2, or 6, the triangle is clockwise. If edge 23 has a delta-direction of 1 (ie, like edge 12), then comparing the slopes of the two edges is unbalanced (both are x-major). If edge 12 has a greater slope, the triangle is clockwise. If edge 23 has a delta-direction of 7 (opposite to edge 12), the slope is compared with the opposite result in terms of whether the triangle is clockwise or counterclockwise.

In all cases of determining proper directivity, the same analysis can be thoroughly applied to all combinations of edge 12 and edge 23 delta-direction. In the balanced case, if the slopes are equal, the triangles deteriorate (i.e. there is no internal space). It can be explicitly inspected and screened out, or, if proper care is taken with respect to numbers, it can be ignored because no pixels are rendered. It is one special case that the triangle splits the view plane, but it may be detected first in the pipeline (eg when front and back clipping is performed).

Note that in most cases, only one side of the triangle is rendered. Thus, if the directivity of the triangle determined by the colorimetric is to be rejected, the triangle can be screened out (ie, no further processing can be assumed without generating a sample). Note that this directivity determination only utilizes one additional comparison beyond the already calculated factors (ie, edge 12 slope and edge 23 slope). Many conventional approaches allow for the use of more complex calculations (through previous set-up calculation steps).

Generating output pixel values from sample values-Figure 19

19 is a flow diagram of an example of a method of selecting and filtering a sample stored in a sample buffer 162 that is super-sampled to produce an output pixel. In step 250, a stream of memory bins is read from the sample buffer 162 which is super-sampled. In step 252, the sample-pixel computing unit 170 may be stored in one or more caches 176 to allow easy access during the convolution processor (step 252). In step 254, memory bins are checked (254) to determine if they contain samples that contribute to the output pixel that are currently being generated. Then, each sample in a bin that can contribute to the output pixel is individually checked to determine if the sample actually contributes (steps 256-258). This determination can be made based on the distance between the sample and the center of the output pixel to be produced.

In one example, the sample-pixel calculation unit 170 is configured to calculate this distance (ie, the range or envelope of the filter at the sample location) and use it to index the table that stores the filter weights according to the filter range. It may be (step 260). However, in another embodiment, a potentially expensive calculation for determining the distance from the center of the pixel (typically including the square root function) to the sample is consistent with the index of the table of filter weights. It can be avoided by using distance. Alternatively, the functions of x and y can be used instead of functions that rely on distance calculations. In one example, this can be done by utilizing a floating point format of distance (eg, 4 or 5 bits of mantissa and 3 bits of exponent), thereby compensating for an increased range of values while maintaining much accuracy. Can be. In one example, the table may be executed in a ROM. However, a RAM table can also be used. Advantageously, using a RAM table, in some embodiments, the graphics system can change filter coefficients on a per-frame basis. For example, the filter coefficients can be changed to compensate for known deficiencies of the display or for the user's personal preference. In some embodiments, the use of a RAM table allows the user to select different filters (eg, through sharpness control on a display device or through a control panel of a window system). Numerous different filters may be performed to produce the desired level of sharpness according to different display configurations. For example, the control panel may have one optimized setting for the LCD display and another optimized for CRT display. The graphics system can also change the filter coefficients on a screen area basis or on a per-output pixel basis within the frame. In another embodiment, specialized hardware (eg, multipliers and adders) can be used to actually calculate the desired filter weight for each sample. The filter weights for samples outside the limits of the convolution filter may simply be multiplied by a filter weight of zero (step 62), or they may be removed completely from the calculation.

Once the filter weight for the sample is determined, the sample can be multiplied by the filter weight (step 264). The weighted sample can then be added to the current total to determine the color value of the final output pixel (266). The filter weight may also be added to the current total pixel filter weight (step 268), which is used to normalize the filtered pixels. Normalization advantageously prevents filtered pixels (eg, pixels with more samples than other pixels) from appearing too bright or too dark by compensating the gain made by the convolution process. After all contributing samples have been weighted and added, the total pixel filter weight can be used to divide the gain by filtration (step 270). Finally, the normalized output pixel can be output and / or processed through one or more of the following processes (not necessarily in this order) before it is finally displayed: gamma correction, Color search using a pseudo color table, direct color, inverse gamma correction, programmable gamma encoding, color space conversion, digital-to-analog conversion (step 274).

In some embodiments, the graphics system may be configured to use the alpha information of each sample to generate a mask that is output with the sample. Masks are used to perform the blue screen effect of soft edges in real-time. For example, a mask can be used to indicate which part of the rendered image (and how much) should be masked. This mask can be used for smooth blue screen effects or ghost effects (e.g., anti-aliased with respect to overlapping areas of two images) (e.g., to partially overlap a transparent object onto another object, scene, or video stream). It may be used by a graphics system or external hardware to blend the rendered image with another image (eg, a signal from a video camera).

Example of Output Pixel Convolution-Figure 20

20 shows a simplified example of output pixel convolution. As shown in the figure, four bins (288A-D) contain samples that may contribute to the output pixel. In this example, the center of the output pixels, i.e., the central place where the filter is applied to produce each output pixel, is located at the boundaries of the bins 288A-288D. Each bin contains 16 samples, and an array of four bins (2 x 2) is filtered to produce an output pixel. Assuming a circular filter is used, the distance of each sample from the pixel center determines which filter value is applied to that sample. For example, sample 296 is relatively close to the pixel center, so that it falls within a filter region having a filter value of eight. Similarly, samples 294 and 292 correspond to filter areas having filter values of four and two. However, sample 290 is outside the maximum filter range, so it has a filter value of zero. Hence sample 290 will not contribute to the value of the output pixel. This type of filter will contribute the most to the sample located closest to the pixel center, while pixels further away from the pixel center will contribute less to the final output pixel value. This type of filtration automatically performs anti-aliasing by smoothing out sudden changes in the image (eg, from black lines to bright backgrounds). Another particularly useful type of filter for anti-aliasing is the windowed sinc filter. Advantageously, the window sync filter has negative protrusions that sharpen portions of the image that have become mixed or “obscured” again. Negative protrusions are filter regions that allow samples to be subtracted from the computed pixel. In contrast, the samples on either side of the negative protrusion are added to the computed pixel.

Example values for samples 290-296 are shown in boxes 300-308. In this example, each sample includes red, green, blue, and alpha values in addition to the location data of that sample. Block 310 represents computing each pixel component value for an unnormalized output pixel. As indicated by block 310, a potentially undesirable gain enters the final pixel value (ie, an output pixel with a red component value of 2000 is far more than any of the red component values of that sample). High), as mentioned above, the filter value may be added to obtain a normalization value 308. The normalization value 308 is used to divide the unnecessary gain from the output pixel. Block 312 shows this process and the final normalized example pixel value.

It is to be noted that the values used herein are for selection purposes only and are not intended to be limiting. For example, filters can have many different areas with different filter values. In one example, some regions may have negative filter values. The filter utilized may be a continuous function whose value is obtained for each sample based on the distance of the sample from the pixel center. Also note that floating point values may be used to increase accuracy. Various filters, such as boxes, tents, cylinders, cones, Gaussians, Kathmull-Rom, Mitchell and Nedvalali filters (Mitchell and Netravallifilter, windowed sinc, etc. may be utilized.

Filter weights do not necessarily need to be powers of two as shown in the figure. The examples shown in the figures are simplified for explanation. A table of filter weights can be used (eg, with a number of entries indexed according to the distance of the sample from the pixel or filter center). Moreover, in some embodiments each sample of each bin may be added to form a pixel value (although some samples in the bin have a weight of zero and thus no contribution to the final pixel value).

Full-Screen Anti-aliasing

Many 3D graphics systems currently only provide real-time anti-aliasing for lines and dots. Some systems can "fuzz" the edges of polygons, but this technique typically works best if all polygons are presorted in depth. This counters the purpose of general-purpose 3D rendering hardware for most applications (that do not pre-sort polygon depths). In one example, graphics system 112 may be configured to perform full-screen anti-aliasing by stochastic sampling up to 16 samples per output pixel, filtered with a 5 × 5-convolution filter.

Variable-Resolution Super Sampling-Figures 21-25

Referring now to FIG. 21, one possible diagram for dividing the sample buffer 162 is shown. In this embodiment, the sample buffer 162 is divided into three regions, the central region 354, the middle region 352, and the peripheral region 350. Each of these areas has an outer boundary in the form of a rectangle, while the middle and peripheral areas have rectangular holes in their centers. Each region may be composed of special (per frame) persistent properties, such as a continuous density of sample density and a continuous size of pixel bins. In one example, the overall density range may be 256. That is, one region can support between one sample per 16 screen pixels (4 × 4) and 16 samples for one screen pixel. In one example, the sample density traverses each region, linearly or nonlinearly. Note that in other embodiments, the display may be divided into a plurality of persistent sized regions (eg, 4 x 4 pixels in size or 40 x 40 pixels in size).

In order to perform operations for polygons that include one or more corners of the region (eg, corners of the central region), the sample buffer may be further divided into a plurality of subregions. In FIG. 21, an example of the sample buffer 162 is divided into sub-regions, as shown. Each of these sub-areas is rectangular and allows graphics system 112 to switch from a 2D address with sub-areas to a linear address in sample buffer 162. Thus, in some embodiments each sub-region has a memory base address indicating where storage for pixels within the sub-region begins. Each sub-region also has a "stride" parameter associated with its width.

Another partition of the super-sampled sample buffer is circular. Referring now to Figure 22, one such example is shown. For example, each region may have two radii (ie 360-368) in that regard, which divides the region into three concentric-regions. The circle-areas may all be centered on the same screen point, ie the center center-point. However, note that the center center-point does not always need to be centered in the center area. In some embodiments, it may be disposed outside the screen (ie, outside the visible display surface of the display device). The embodiment shown supports seven unique circle-regions. Some of the circles can be divided across two different regions, thereby reducing the unique circular-region to five or less.

Circular regions can outline regions of continuous sample density that are actually used. For example, in the embodiment shown in the figure, the center region 354 can assign a sample buffer density of 8 samples per screen pixel, but outside the innermost circle 368, it only takes 4 samples per pixel. Then outside of circle 366 it can use only two samples per pixel. Thus, in this embodiment the rings do not necessarily save real memory, but they can potentially save memory bandwidth into and out of the sample buffer (as well as pixel convolution bandwidth). In addition to indicating the sample density of different effects, the rings can also be used to indicate the different sample localizations used. As mentioned earlier, this sample location plot can be stored on a chip of RAM / ROM or in a programmable memory.

As discussed above, in some embodiments the super-sampled sample buffer 162 may be further divided into bins. For example, a bin may store a single sample or an array of samples (eg, 2 × 2 or 4 × 4 samples). In one example, each bin can store 1 to 16 sample points, although other configurations are possible and contemplated. Each region can be configured with a special bin size and continuous memory sample density. Note that lower density regions do not necessarily have a larger bin size. In one example, an area (or at least an interior area) is a multiple of the exact integer of the bin size that contains the area.

Variable-resolution super-sampling involves computing various numbers of samples for each pixel displayed on the display device. Certain areas of the image (eg near the edges of an object) can benefit from having more samples, while other areas (eg, smooth areas with constant color and brightness) require other samples. You can't. To save memory and bandwidth, special samples may be used only in areas that can benefit from increased resolution. For example, if a portion of the display is painted in a constant blue color (e.g. as a background), it will not be particularly useful because all special samples simply have a constant value (same as the background color being displayed). In contrast, if the second region on the screen is displaying an object with complex textures and edges rendered in 3D, using additional samples would be useful to avoid special artifacts such as aliasing. Many different methods can be used to determine or predict which regions of an image can benefit from higher sample densities. For example, edge analysis may be performed on the final image, and the information is used to predict how the sample density should be distributed. In addition, the application of software will be able to indicate which regions of the frame have been assigned higher sample densities.

Numerous different methods can be used to perform variable-resolution super-sampling. These methods tend to fall into two general categories: (1) methods related to the draw or rendering process, and (2) methods related to the convolution process. For example, by using any of the following methods, samples may be rendered into the sample buffer 162 being super-sampled.

1) uniform sample density;

2) variable sample density (eg, middle, center, and periphery) on a per-region basis;

3) sample density that is variable by varying the density on a scan-line basis (or on a small number of scanline basis).

Variable sample density on a scan-line basis can be achieved by using a density lookup table. For example, a table can specify that the first five pixels of a particular scanline each have three samples, that the next four pixels each have two samples, and so on.

On the convolution side, the following methods are possible:

1) uniform convolution filter;

2) continuously variable convolution filters;

3) Convolution filters that operate at multiple spatial frequencies.

The uniform convex filter, for example, has a continuous range (or number of samples selected) for each computed pixel. In contrast, a continuously variable convolution filter is used to determine the number of samples used to compute a pixel. The number can change gradually. Its function can vary continuously from the maximum at the center of interest to the minimum at the surrounding area. Other combinations of this method are also possible (for both the rendering side and the convolution side). For example, a continuously variable convolution filter can be used for the sample, while a constant sample density can be used for the render side.

Different methods are also contemplated for determining which region of an image to allocate more samples per pixel. In one example, if the image on the screen has the main focus (e.g., a character such as Mario in a computer game), a larger number of samples may be computed for the area near Mario, and other areas (e.g. near the background or on the screen). Fewer pixels may be computed for pixels near edges.

In another embodiment, the observer's center point can be determined by eye / head / hand-tracking. In the head-tracking example, the direction the observer sees is determined or predicted from the orientation of the observer's head, which can be measured using various mechanisms. For example, in order to provide orientation information to the graphics system 112, a helmet or helmet (with eye / head tracking) worn by the observer may be used alone or a hand-tracking mechanism, fat or eye-tracking sensor may be used. There will be. Another alternative example includes the use of a head-tracking, or head-tracking sensor and / or eye-tracking sensor using an infrared reflecting point already attached to a user. One method of using head tracking and hand tracking is described in US Patent No. 5,446,834 (Method and Apparatus for High Resolution Virtual Reality Systems Using Head Tracked Display, inventor Michael Deering, published 29 August 1995). Is shown. Other methods for head tracking are also possible and contemplated (e.g., infrared sensors, electromagnetic sensors, capacitive sensors, video cameras, sound and ultrasonic sensors, sensors attached to clothes, video tracking devices, conductive inks, pressure gauges, Power-feedback detector, fiber optic sensor, air sensor, magnetic tracking device, mechanical switch).

As mentioned earlier, eye-tracking is particularly beneficial when used in conjunction with head-tracking. In an embodiment for eye-tracking, the viewing direction of the observer can be measured directly by sensing the orientation of the observer's eye relative to the observer's head. When combined with other information relating to the position and orientation of the observer's head with respect to the display device, this information will enable accurate measurement of the observer's center point (or center points when two eye-tracking sensors are used). One possible method for eye tracking is US Pat. No. 5,638,176 (named “Inexpensive Interferometer Eye Tracking System”). Other methods for eye tracking are also possible and contemplated (eg, methods for head tracking listed above).

Regardless of which method is used, the observer's center point changes position and so is the distribution of the sample. For example, if what the viewer sees is centered in the upper left corner of the screen, the pixels corresponding to the upper left corner of the screen are allocated 8 or 16 samples, respectively, while the pixels in the opposite sphere are pixels. Only one or two samples will be allocated per. Once the observer's gaze changes, so does the allocation of samples per pixel. If the observer's gaze moves to the bottom right on the screen, the pixel in the top left corner on the screen will be assigned only one or two samples. Thus, the number of samples per pixel can vary actively with different regions of the screen in relation to the observer's center point. In some embodiments, multiple users may each have a head / eye / hand tracking mechanism that provides input to graphics system 112. In this embodiment, two or more center points on the screen can be imagined, with corresponding areas of high and low saple density. As mentioned earlier, these sample densities can affect the rendering process only, the filtration process only, or both processes.

Referring now to FIGS. 24A-B, an example of a method of allocating the number of samples per pixel is shown. The method allocates the number of samples according to the locations of the pixels relative to one or more center points. In FIG. 24A, an eye-tracking or head-tracking device 360 is used to determine the center point (ie, the focal point that the observer looks at) 362. This is determined by using a tracking device 360 to determine the direction facing the observer's eye (shown as 364 in the figure). As the figure shows, in this embodiment, the pixel is divided into a central region 354 (centered around the center point 362), an intermediate region 352, and a peripheral region 350.

Three sample pixels are indicated in the figure. The sample pixel 374 is disposed inside the center region 314. Assume the center region 314 has eight samples, and if each pixel touches four bins, then up to 32 samples will contribute to each pixel. The sample pixel 372 is disposed inside the middle region 352. If the middle region 352 consists of bins with four samples, and the convolution radius for each pixel touches four bins, then up to 16 samples will contribute to each pixel. The sample pixel 370 is disposed in the peripheral area 350. If the peripheral region 370 consists of bins with one sample each, and the convolution radius for each pixel encounters one bin, then one pixel per sample is correlated with respect to the pixels in the peripheral region 350. There is a relationship. These values are merely examples, and other numbers of regions, samples per bin, and convolution radius may be used.

Referring now to FIG. 24B, the same example is shown, but the center point 362 is different. As the figure shows, when the tracking device 360 detects a change in the position of the center point 362, it provides input to the graphics system, which adjusts the position of the center region 354 and the middle region 352. . In some embodiments, some of the regions (eg, the middle region 352) may extend beyond the edge of the display device 84. In this example, pixel 370 is now within center point 354, while pixel 372 and pixel 374 are in the surrounding area. Assuming the configuration of the sample as in the example of FIG. 24A, up to 32 samples may contribute to pixel 370, while only one sample will contribute to pixel 372 and pixel 374. Advantageously, this configuration can allocate more samples for the area near the center point (ie, the focal point of view from the viewer). This can provide the viewer with a more realistic image without having to strand a large number of samples for every pixel on the display device 84.

Referring now to FIGS. 25A-B, another embodiment of a computer system consisting of a sample buffer being variable resolution super-sampled is shown. In this embodiment, the center of attention of the observer is determined by the position of the main character 362. As the hero 362 moves around the screen, the middle and center regions are centered around the hero 362. In some embodiments, the main character may be a simple cursor (eg, moved by keyboard input or mouse).

In yet another embodiment, an area with a higher sample density may be centered around the center of the screen of display device 84. Advantageously, this can reduce the control requirements of software and hardware while providing a sharper image at the center of the screen (which is the focus of the observer's attention for most of the time).

Although the above embodiments have been described in considerable detail, other variations are possible. Once the above description is fully understood, it will be apparent to those skilled in the art that numerous changes and modifications can be made. The scope of the following claims is to be construed as including all such changes and modifications. Note that the headings used herein are for organizational purposes only and are not intended to limit the scope of the appended claims or the description of the invention provided herein.

While this specification and the accompanying drawings are to be understood and understood by those skilled in the art, the present systems and methods described herein may be characterized by graphics systems and subsystems, computers, computing devices, set-top boxes, game consoles, Personal digital assistants (PDAs), digital television, video processors, graphics processors, multimedia systems and processors, virtual reality systems and other rendering systems and / or display graphics data, and the like Can be.

Claims (22)

One or more processors (352) for receiving a set of graphic data and rendering a plurality of samples based on the graphic data; A sample buffer 162 configured to store the plurality of samples; A plurality of sample-pixel calculation units 360, wherein the sample-pixel calculation unit is configured to receive and filter samples from the sample buffer to produce output pixels, the output pixels being used to form an image on a display device. And wherein each of the plurality of sample-pixel operation units comprises a sample-pixel operation unit (360) configured to generate a pixel corresponding to a different one of the plurality of regions of the image. A method of rendering a set of graphic data, Receiving the graphic data; Generating one or more samples based on the graphic itter; Storing the sample; Dividing the sample into a plurality of regions; Selecting a storage sample from the plurality of areas; Filtering the selected samples in parallel to form a plurality of output pixels that can be used to form an image on a display device. The method of claim 2, Each sample comprises a color component, The filtering step includes determining which sample is within a predetermined filter envelope; Multiplying the sample within the predetermined filter envelope by one or more weighting factors, the weighting elements varying with the position of the sample relative to the center of the filter envelope; Summing the weighted samples to produce an output pixel. A display device 84; Means for receiving a set of graphical data; One or more processors (352) configured to render a plurality of samples based on the set of graphic data; A sample buffer 162 configured to store the rendered sample; A plurality of sample-pixel calculation units 170 configured to filter the stored sample to produce an output pixel, wherein the output pixels can be used to form an image on a display device, each of the plurality of sample-pixel calculation units Is configured to generate pixels corresponding to one of a plurality of different regions of the image. The method according to any one of claims 1 to 4, Each said sample comprises a color component; The sample-pixel calculation unit determines which sample is within a predetermined filter envelope, and multiplies a sample within the predetermined filter envelope by one or more weighting factors, wherein the weighting factor is determined by the weight of the sample relative to the center of the filter envelope. A system or method that varies with position and sums the weighted samples to form one or more output pixels. The method according to any one of claims 1 to 5, Wherein each area corresponds to a different vertical or horizontal row of said image. The method according to any one of claims 1 to 5, Wherein each region comprises portions of an image corresponding to one or more odd or even scanlines. The method according to any one of claims 1 to 5, Wherein each region comprises different quadrants of the image. The method according to any one of claims 1 to 8, The system or method overlaps the plurality of regions. The method according to any one of claims 1 to 9, The display device includes a plurality of personal display devices, Each area corresponding to a single one of the plurality of personal display devices. The method according to any one of claims 1 to 9, The display device includes a plurality of personal display devices, Wherein each region corresponds to a different one of the plurality of personal display devices. The method according to any one of claims 1 to 11, The region or system varies in dimension of frame-to-frame reference. The method according to any one of claims 1 to 11, Said region of said image varies in dimension of a frame-to-frame reference to balance the number of samples filtered by each of said sample-pixel computing units. The method according to any one of claims 1 to 5 and 9 to 13, Wherein each area is a different rectangular portion of the image. The method according to any one of claims 1 to 14, Each sample comprises a color component, The sample-pixel calculation unit determines which sample is within a predetermined filter envelope; A system or method configured to multiply a sample within the predetermined filter envelope with one or more weighting factors that vary with the position of the sample relative to the center of the filter envelope. The method according to any one of claims 1 to 15, Wherein each sample comprises a z-component. The method according to any one of claims 1 to 16, The sample stored in the sample buffer is a double buffer. The method according to any one of claims 1 to 17, The area or system wherein the size changes over time. The method according to any one of claims 1 to 18, Wherein each region varies in size on a frame-to-frame basis to balance the number of samples filtered by the sample-pixel computing unit. The method according to any one of claims 1 to 19, Wherein each region varies in size on a frame-to-frame basis to equalize the number of samples in each region. The method according to any one of claims 1 to 20, The sample stored in the sample buffer is stored in a bin (bin). The method according to any one of claims 1 to 21, The system or method wherein the number of samples filtered for each pixel varies from region to region.