KR101739432B1 - Zone-based tone mapping - Google Patents

Abstract

The present invention relates to a method of tone mapping high dynamic range images for display on a low dynamic range display, wherein a high dynamic range image is first accessed. High dynamic range images are divided into different areas, with each area appearing as a matrix, where each area of the matrix is a pixel's weight or probability. The exposure of each region is determined or calculated, and the exposure value is applied to the region in response to a weight or probability. Then, the different regions are fused together to obtain the final tone mapped image.

Description

ZONE-BASED TONE MAPPING < RTI ID = 0.0 >

This application claims priority to U.S. Provisional Patent Application No. 61 / 269,760, filed June 29, 2009, the entirety of which is incorporated herein by reference.

The present invention relates to tone reproduction of high dynamic range (HDR) content on a low dynamic range (LDR) display, which is also known as a tone mapping problem. In particular, at least one embodiment includes (1) automatically generating a displayable LDR image from HDR data that is consistent with a human perception of the HDR scene, and (2) providing user-friendly control for manual adjustment ≪ / RTI >

The tone mapping problem relates to tone reproduction of high dynamic range (HDR) content on a low dynamic range (LDR) display. In most applications, the tone mapping process has to satisfy mainly two requirements: maintaining image detail, e.g., local contrast, and maintaining a relative brightness appearance. The currently known work on tone mapping focuses on the first requirement, and the second most important requirement from the artist's perspective is just neglect. In addition, currently available tone mapping algorithms do not allow to manipulate the tones of different parts of the picture and thus often fail to meet the sense of the original HDR content.

High dynamic range (HDR) has recently received attention as an alternative format for digital imaging. The traditional low dynamic range (LDR) image format is designed for displays conforming to ITU-R Recommendation BT 709 (a.k.a.Rec. 709), where only a few hundred times the magnitude of the dynamic range can be achieved. However, actual scenes have a dynamic range that is about a few tens of billion times larger during the day, and a human visual system (HVS) can perceive hundreds of thousands of times at the same time.

The amount of visual content available in the HDR format is increasing: recent advances in digital sensors and motion picture films allow content creators to capture images with a very high dynamic range and use computer generated graphics (e.g., Animated movies, visual effects, and games) allow you to create visual content with virtually unlimited dynamic range. However, HDR displays are not yet mainstream devices; Although several HDR devices are already available in prototype and top-of-the-line HDTV, the number of such displays is still very small compared to widely used LDR displays.

To display an HDR image on an LDR display device, a tone mapping method is used to map an HDR image available with radiance to an 8-bit RGB index number. The tone mapping process is not obvious because the tone mapped LDR image must simulate the processing that occurs at the HVS, so that the HVS believes it is close enough to the original HDR image. This process requires the tone mapping algorithm to maintain both local contrast and perceptual brightness.

Recently, tone mapping of HDR images has been studied not only in computer graphics but also in the image / video processing community. Approximately, the tone mapping method can be divided into two main categories: global tone mapping and local tone mapping.

Global tone mapping uses global curves to map the intensity to image intensity. Global tone mapping has advantages such as low complexity and easy manual control, but it can not keep all of the details when it is very high dynamic range. Therefore, global tone mapping is not suitable for applications requiring very high quality output (such as post-production).

On the other hand, the local tone mapping method provides a higher quality result by compressing each individual pixel according to the local image feature. In particular, these methods attempt to simulate the visual adaptation that occurs in HVS, but in practice most methods do not explicitly mimic the behavior of HVS. Instead, these methods make simple assumptions about HVS and then use these assumptions to try to compress the dynamic range of the image, resulting in a visually good appearance. Through careful fine-tuning of the local tone mapping method, it is possible to produce convincing results for a relatively large area of the HDR image, but still the understanding of visual adaptation is still far from perfect It is far. Therefore, there is no algorithm that behaves like a human eye. In addition, these methods do not provide good manual control of the tone mapping process and typically severely limit the creativity involved in the tone correction process.

Tone mapping was studied by image processing researchers as well as painters and film photographers. They face the same problem of using a limited dynamic range medium (i.e., a canvas for a painter, a print for a photographer) to represent high dynamic range scenes. Referring to Figure 1, here we review the "zone system (100)", a photographic technique formulated by Ansel Adams and Fred Archer. The zoning system assigns numbers from 0 to 10 to different perceptual brightness, where 0 represents black, 5 represents mid-gray, and 10 represents pure white. These values are known as zones. In the theory of the zoning system, the photographer first identifies the main elements in the scene and places them on the desired area.

This treatment depends on the perception of the scene rather than the measurement of the brightness. Then, a light meter is used to measure the brightness of each major element in the scene. Since only a single exposure value per shot may be present, the exposure value is selected such that the most important factor is mapped to the desired zone. As a result, other (and important) elements may be mapped to "false" zones, becoming too dark or too bright. Then, in the printing process, this problem is corrected by applying a "dodge and burn" operation, a printing technique in which some light from a portion of the print is suppressed or damped during development, More light is added to the light. Therefore, the main element that is mapped to the lower zone relative to the desired zone is exposed to light longer than the rest of the image. Similarly, key elements that are mapped to higher zones than the desired zone are less exposed. This local processing ensures that the main elements of the picture are mapped to the desired area at the final output. In other words, the perceived brightness of these key elements corresponds to how these key elements actually look.

While this approach can be used for digital images, there is no way to have good performance in the automatic mode and at the same time provide intuitive control in the user-assisted mode.

A method for tone mapping high dynamic range images for display on a low dynamic range display is provided, wherein a high dynamic range image is first accessed. Then, the high dynamic range image is divided into different regions, each region appearing as a matrix, where each element of the matrix is a weight or probability of a pixel. The exposure of each region is determined or calculated, and the exposure value is applied to the region in response to a weight or probability. The different regions are then fused together to obtain the final tone mapped image. The method may further comprise identifying or constructing a different perceptual brightness for a high dynamic range image, or for a final tone mapped image. In addition, the method may comprise any of the following steps: determining an area responsive to luminance data, each constructing an anchor value to build one of the areas, Channel. The tone mapping may be performed based on the luminance channel, and thereafter, applied to the color channel through post-processing, and tone mapping may be performed based on each color channel.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: Fig.

The present invention has the advantage of having good performance in the automatic mode and at the same time providing user-friendly control in the user-assisted mode.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a diagram of a known area system scale.
Figure 2 is a flow chart of a zone-based tone mapping method in accordance with the present invention;
Figure 3 is a flow chart of an application of the method of Figure 2 to a color HDR image.
Figure 4 is a flow chart of another application of the method of Figure 2 for a color HDR image.
Figure 5 is a flow chart illustrating tone correction of an LDR image using the method of Figure 2;
Figure 6 shows a pair of sample images in which one image is enhanced according to the method of the present invention;

Similar to the concept of a "zone system" developed in film photography for a typical passive tone mapping, the method generally involves four steps:

a. Identifying a key element in the image,

b. Mapping each major element to a zone, respectively,

c. Measuring the brightness of each key element,

d. Determining a global exposure value, and

e. Dodge and step in the printing process to ensure that each key element is within the correct area of the final print

≪ / RTI >

To describe the method in more detail, we can first define the input and output of the tone mapping problem. First, it is assumed that the input is the scene's brilliance in a known color space with a known primary color. When the HDR data is not calibrated, the brilliance data may be an absolute brilliance or a linearly scaled brilliance. The output is a tone mapped image,

The luminance image can be calculated from the HDR data. If the HDR data is in the XYZ color space, the Y component can be used as the luminance image. HDR data is recorded in Rec. If you use the same primary color as 709, the conversion from RGB color space is:

. ≪ / RTI >

Other conversions between RGB (or other color space) and luminance images can be used depending on the format of the input image.

It then defines the simplest form of tone mapping: single exposure. Without losing generality, it is assumed that the HDR data has only one color channel, which can be a luminance image. Through linear scaling, anchor points are defined. A pixel will be saturated and mapped to 1 if the luminance of this pixel exceeds the anchor point, otherwise the pixel will be mapped to a value of 0 to 1. Therefore, the linear scaling of a single exposure is:

Where A is an anchor point and S (x) is an anchor point,

, Where rho typically takes values in the region [2.2, 2.4] and represents the gamma of the output device (where the tone mapped image appears).

The resulting image I can be quantized and displayed on a conventional LDR display. Note that other qualifications of S (x) may be possible: for example, instead of a power function, an S-shaping curve may be used. Generally, any global mapping curve can be used for S.

The zone system then applies to digital tone mapping. As shown in Fig. 2, the main steps are as follows. The high dynamic range (HDR) image of the input is first divided into different regions in step 10. This may be a hard partition or a fuzzy partition. In any case, each region can be represented by a matrix, where each element of the matrix is a probability (weight) of the pixel. If a clear division is used, the image pixel belongs to a single area, so the probability is 0 or 1. If unclear partitions are used, the probability can be any value between 0 and 1, as each pixel can be distributed over several (even entire) areas.

Then, at step 12, the algorithm determines which zone each zone is mapped to. This inherently estimates the exposure for each area. The mapping between regions and zones can also be done through interaction with a user by providing a suitable user interface.

Then, at step 14, each area is exposed through an exposure parameter of the area.

The final tone mapped image is then generated by fusing together the different regions (each exposed through the exposure value of the region) in step 16, using the weights obtained in step 10 Fusion or mixing treatment is used.

Optionally, in steps 18 and 20, the user can examine the appearance of the tone mapped image, change the exposure value of one or more areas, and then change the parameters in the step until the result is satisfactory , Through steps 14 through 18, via a suitable user interface.

For video tone mapping, processing for one main frame in a scene may be performed, and the same parameters may be applied to all frames of the scene.

Embodiments of each of the major steps are described in greater detail below.

In step 10,

The goal of partitioning is to divide the image into regions, each containing an object that must be mapped to the same region. That is, each area must require a single exposure. Partitioning can be done in various ways using various image processing techniques. Here, a simple but efficient approach is described. First, the luminance image is calculated from the HDR luminance data. The division is performed only on the luminance image. The average, maximum and minimum brightness of the image are as follows:

Where R _min and R _max are the two predefined percentages, max _{R (X)} is the smallest value of X among R% of the values in X, and min _{R (X)} is the R It is the largest value of X among%.

As discussed above, within each region, the pixels must have the same exposure. A series of anchor points A _i (i = 1., N) is defined such that each A _i defines a region and is also used to generate a single exposure image.

In this embodiment, the anchor point is:

, Where E is a constant and may take a value of, for example, 8. The number N of regions in the above equation is:

, Which can cover all of the luminance region.

It is easy to observe that the distance between two neighboring anchor points is two "stops" for photography.

Once the anchor points of each region are known, the weight of each pixel is calculated for each region. In general, for each area {defined by the corresponding anchor point A _i }, the value of the closest pixel in the single exposure image is 0.5, and this area {defined by the corresponding anchor point A _i ) The weight of this pixel increases.

Thus, the weight of the pixel at position (i, j) to the area n {defined by the anchor point A _n ) is:

, Where C is the normalization coefficient, C is:

.

The calculated weight takes a value in the area [0, 1] and thus defines an unclear partitioning into the N areas of the luminance image. This means that although some of the N regions may have large weights, each region may contain all of the pixels in the image.

In other implementations, the weights are binarized (i.e., weights are 0 or 1), resulting in a clear partition:

Note that the weight (W _n ) as well as the anchor point (A _n ) are fixed once the segmentation is done. In the next section, it is observed that the exposure for each region can be adjusted while the weights remain unchanged.

Estimation of exposure at step (12)

Once each region is divided, each region is mapped to a region. That is, the anchor points are defined for each region so that after a single exposure, each region can be exposed appropriately.

It is a very subjective task to determine which areas each area should be mapped to, because it depends on how the HVS completes the visual adaptation. In conventional zone systems, mapping of key elements to zones is visually determined by the photographer.

A number of different algorithms can be used to estimate the exposure of each region. In a simple implementation, all regions are mapped to a neutral gray, after which the user can change the anchor point interactively. This means that the estimated exposure is the value of the anchor point equal to the anchor point value used to define the region:

Here,? _N is a variable that can be changed by the user.

The default value for λ _n is 0, but the method allows you to manually modify this default value to reach the desired appearance.

Applying exposure parameters in step (14)

Once the user has a partition of the HDR image and an anchor point for each area, the corresponding LDR image can be generated from the HDR data using the estimated exposure:

.

.

In step 16, image fusion and enhancement

The goal of this step is to mix all of the areas (each exposed through the exposure parameters) together. Several fusion methods may be possible, some of which are described below.

Image Fusion - First Embodiment

Calculate the weighted average of the LDR image to generate the tone mapping result (T).

This is a low complexity method. Unfortunately, this method is very sensitive to image weights, resulting in visual artifacts in most implementations.

Image Fusion - Alternative Embodiment

A more sophisticated fusion process is to combine these LDR images. Other image fusion methods follow a multi-resolution approach using pyramids. Although this method is more complex, it is quite robust to weighting (i.e., partitioning of regions into regions), resulting in almost seamless transition between regions.

color HDR  Tone of image Mapping

FIG. 3 illustrates how the proposed region-based tone mapping approach 300 can be used to tone-map a color HDR image. First, if it is assumed that the color HDR image is in the RGB color space, the luminance image is calculated using the equation given above. The luminance image is then processed according to the framework of the described method. Finally, the color processing step applies tone mapping of the luminance image to each color component. In a particular implementation, the color processing step resizes each pixel of each color component to the same amount, where the corresponding pixel of the luminance image is scaled, and then performs gamma correction and quantization. This processing can be performed using the following equation:

, Where Q (.) Denotes the quantization function, where gamma is the gamma of the output device.

Tone correction of a color LDR image can be accomplished using the region based tone mapping method described above. These methods can be applied to automatically or manually correct LDR images. As shown in process flow 500 of FIG. 5, a further step compared to the processing of HDR images is the conversion from LDR to HDR, which can be done using inverse quantization and inverse gamma conversion:

.

.

Some modifications to this step may be possible. For example, tone mapping may be performed independently for each color component, instead of using a luminance image, as best shown in FIG. 4 (e.g., red mapping flow 401, green mapping Flow 402 and blue mapping flow 403). In addition, tone mapping may be performed on the color component instead of using the luminance image.

In another variation, a single component (e.g., one of a color component or a luminance image) may be used in some of the steps and the color component may be used in another step. For example, luminance may be used for steps 10-12 and color components may be used for steps 14-16.

The region-based tone mapping method described herein results in improved display of the image, especially in the conversion from HDR image to LDR image. An example of these results is shown in FIG. 6, where the right image 601 was processed according to the method described above, and the left image 602 was not processed in the manner described above. Here, the enhanced tint can be observed in the sky and ground portions of the image.

[0064] Now, alternative embodiments having one or more implementations having particular features and aspects are described, but the features and aspects of the described implementations may also be adapted to other implementations.

For example, these implementations and features may be used as a background for coding video and / or coding other types of data. In addition, these implementations and features may be implemented in the H.264 / MPEG-4 AVC (AVC) standard, the AVC standard with MVC extension, the AVC standard with SVC extension, the 3DV standard and / , Or as a background not accompanied by a standard,

In addition, implementations may use various techniques to send signals, including, but not limited to, SEI messages, slice headers, other high level syntax, high level syntax, out of band information, data stream data and implicit signaling can do. Thus, while the implementations described herein may be described in a particular context, such description should not be taken to limit features and concepts for such implementations or backgrounds.

Reference in the specification to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" of the principles of the invention, as well as other variations thereof, Structure, characteristic, etc., are included in at least one embodiment of the principles of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in one embodiment" or " in one embodiment " or "in one embodiment ", and any other variation described in various sections throughout the specification, It is not necessary to refer to the embodiment.

The implementations described herein may be implemented, for example, as a method or process, an apparatus, a software program, a data stream, or a signal. Although an implementation is discussed only as a background for a single type of implementation (e.g., discussed only by way of example), the implementation of the discussed features may also be implemented in other forms (e.g., a device or a program). For example, the device may be implemented with suitable hardware, software, and firmware. The method may be implemented, for example, in an apparatus such as a processor, generally referred to as a processing device, which includes a computer, microprocessor, integrated circuit or programmable logic device. In addition, the processor includes, for example, a computer, a cell phone, a communication device such as a personal digital assistant ("PDA"), and other devices that facilitate communication of information between end users.

Implementations of the various processes and features described herein may be implemented in different devices or applications, particularly, for example, equipment or applications related to data encoding and decoding. Examples of such equipment are encoders, decoders, post-processors that process outputs from decoders, preprocessors that provide inputs to encoders, video coders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers , Cell phones, PDAs, and other communication devices. As is evident, the equipment may be mobile and may even be mounted on mobile means.

In addition, the method may be implemented by instructions executed by a processor, and such instructions (and / or data values generated by the implementation) may be stored, for example, in an integrated circuit, a software carrier, Readable medium such as a compact disk, a random access memory ("RAM") or other storage device such as a read only memory ("ROM"). The instructions may form an application program that is explicitly implemented on the processor readable medium. The instructions may be, for example, hardware, firmware, software or a combination thereof. The instructions may be found, for example, in an operating system, a separate application, or a combination of the two. Thus, a processor may be characterized as both a device configured to perform processing and a device comprising a processor-readable medium (such as a storage device) having instructions to perform processing. In addition, the processor readable medium may store data values generated by an implementation in addition to, or instead of, a directive.

As will be appreciated by those skilled in the art, implementations may generate various signals that are formatted, for example, to convey information that may be stored or transmitted. The information may include, for example, instructions to perform the method, data generated by one of the described implementations. For example, the signal may be formatted to convey the rules for writing or reading the syntax of the described embodiment as data, or to convey the actual syntax-value recorded by the described embodiment as data. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum), or a baseband signal. The format may include, for example, encoding the data stream and modulating the carrier through the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over various, different wired or wireless connections, as is known. The signal may be stored on the processor-readable medium.

A number of implementations have been described. However, it should be understood that various modifications may be made. For example, elements of different implementations can be combined, supplemented, modified or eliminated to create different implementations. Furthermore, those skilled in the art will recognize that other structures and processes may be substituted for the disclosed structures and processes, and that the resulting implementations may be implemented, at least substantially in substantially the same manner (s) to achieve the same result (s) Will perform the same function (s). Accordingly, these and other implementations are contemplated by the present disclosure and are within the scope of the present disclosure.

The foregoing describes some of the possibilities for carrying out the present invention. Many other embodiments are possible within the scope and spirit of the invention. It is therefore to be understood that the foregoing description is to be regarded as illustrative rather than limiting, and that the scope of the invention is given by the appended claims along with their full scope of equivalents.

10: Split image into regions
12: Estimate exposure for each area
14: Apply exposure parameters to each area
16: Mixed image fusion with regions
18: Satisfactory results?
20: Adjust the exposure parameters for each area

Claims (14)

CLAIMS What is claimed is: 1. A method of tone mapping an input image to display an input image on a display designed for a dynamic range lower than the dynamic range of an input image data format,
Accessing the input image;
Dividing the input image into a plurality of regions, each region being represented by a matrix having elements corresponding to the pixels of the input image, wherein each element of the matrix includes a pixel corresponding to the element having a weight or probability Dividing the input image into a plurality of regions;
Determining or calculating an exposure parameter that specifies scaling of pixel values within each region;
Scaling pixel values within each region according to the exposure parameters for that region;
Fusing the plurality of regions together to obtain a final tone mapped image; And
Providing a user with a user interface for adjusting the exposure parameters
Includes a way to tone the input image. delete delete 2. The method of claim 1, wherein dividing the input image into a plurality of regions comprises dividing the input image into a plurality of regions according to a luminance of a pixel of the input image. 5. The method of claim 4, further comprising building a series of discrete pixel values into an anchor value, wherein each region of the plurality of regions corresponds to one of the anchor values. 2. The method of claim 1, wherein dividing the input image, determining or calculating an exposure parameter, scaling pixel values, fusing a plurality of regions, and providing a user interface comprises: Wherein the step of performing tone mapping is performed based on a luminance channel, the step of scaling the pixel values and the step of fusing a plurality of regions are applied to all color channels of the input image. 2. The method of claim 1, wherein dividing the input image, determining or calculating an exposure parameter, scaling pixel values, fusing a plurality of areas, and providing a user interface comprises: ≪ / RTI > is performed based on each color channel.
delete delete delete delete delete delete delete

Images