BR112022018207B1 - COMPUTER IMPLEMENTED AUDIO PROCESSING METHOD, NON-TRAINER COMPUTER READABLE MEDIA AND AUDIO PROCESSING APPARATUS - Google Patents

Abstract

An audio processing method includes generating harmonics in a hybrid complex quadrature mirror filter domain. Generation of the harmonics may include multiplication, using a feedback delay circuit and dynamic compression. Harmonics can be generated based on one or more hybrid subbands of the complex transform domain signal.

Description

CROSS REFERENCES OF RELATED ORDERS

[001] This application claims priority to the International PCT Application PCT/CN2020/080460 filed on March 20, 2020; and U.S. Provisional Application 63/010,390 filed on April 15, 2020; all of which are incorporated herein by reference.

FIELD

[002] The present specification refers to audio processing and, in particular, bass enhancement.

FUNDAMENTALS

[003] Unless otherwise indicated in this document, the materials described in this section do not constitute the prior art as the claims in this application and are not admitted to be prior art by inclusion in this section.

[004] The bass effect is a desirable user experience and user evaluation indicator for mobile devices such as cell phones, media players, tablet computers, laptop computers, headphones, earbuds, etc. Due to the physical constraints of transducers in mobile devices (e.g. diaphragm size, magnet weight, etc.), it is challenging for the mobile device speaker to fully reproduce the acoustics of the original bass sound. As a result, mobile devices often implement audio processing techniques (e.g. using software processes, etc.) to improve bass sound. These bass enhancement processes can be broadly referred to as "virtual bass" techniques.

SUMMARY

[005] A problem with existing bass enhancement systems is that they can have high computational complexity. Given the above, there may be a need to implement bass enhancement with reduced computational complexity.

[006] As discussed in more detail in this document, the embodiments discuss techniques for bass enhancement based on the "missing fundamental" principle. This principle psychoacoustically states that if a human hears harmonics of a low-frequency signal rather than the low-frequency (fundamental) signal itself, the listener's brain is able to extrapolate and therefore perceive the missing low-frequency signal. Thus, for speakers that are physically unsuitable for reproducing low-frequency (bass) signals, one way to psychoacoustically improve the quality is to generate harmonics for the low-frequency range to improve the bass effect.

[007] The bass enhancement technique disclosed in this specification is less computationally complex compared to conventional virtual bass technologies, but achieves a similar effect. Therefore, the modalities save computational complexity. Furthermore, the reduced complexity allows for lower latency. The technique may also include audibility adjustment schemes to adjust the power of the generated harmonics, which makes the perception of the resulting audibility more realistic and the bass effect more convincing.

[008] The techniques disclosed in this specification can be used to improve the output of medium-sized speakers and smaller transducers, for example, mobile phone speakers, wireless speakers, etc.

[009] According to one embodiment, a computer-implemented method of audio processing includes receiving a first transform domain signal. The first transform domain signal is a hybrid complex transform domain signal having a plurality of bands. At least one of the plurality of bands has a plurality of subbands and the first transform domain signal has a first plurality of harmonics.

[010] The method further includes generating a second transform domain signal based on the first transform domain signal. The second transform domain signal is generated by generating harmonics to the first transform domain signal according to a nonlinear process. The second transform domain signal has a second plurality of harmonics that differs from the first plurality of harmonics. The second transform domain signal is further generated by performing audibility expansion in the second plurality of harmonics. The second transform domain signal is a complex valued signal having an imaginary part.

[011] The method further includes generating a third transform domain signal by filtering the second transform domain signal. The third transform domain signal has a plurality of bands and at least one of the plurality of bands has a plurality of subbands. The method further includes generating a fourth transform domain signal by mixing the third transform domain signal with a delayed version of the first transform domain signal, wherein a given subband of the third transform domain signal is mixed with a corresponding subband of the delayed version of the first transform domain signal.

[012] According to another embodiment, an apparatus includes a speaker and a processor. The processor is configured to control the apparatus to implement one or more of the methods described herein. The apparatus may additionally include details similar to those of one or more of the methods described herein.

[013] According to another embodiment, a non-transitory computer-readable medium stores a computer program that, when executed by a processor, controls an apparatus to perform processing including one or more of the methods described herein.

[014] The following detailed description and accompanying figures provide further understanding of the nature and advantages of various implementations.

BRIEF DESCRIPTION OF THE DRAWINGS

[015] FIG. 1 is a block diagram of an audio processing system 100.

[016] FIG. 2 is a block diagram of a bass boost system 200.

[017] FIG. 3 is a block diagram of a 300 harmonic generator.

[018] FIG. 4 is a block diagram of a 400 harmonic generator.

[019] FIG. 5 is a block diagram of a 500 harmonic generator.

[020] FIG. 6 is a graph 600 showing equal audibility curves.

[021] FIG. 7 is a graph 700 showing various compression gains c.

[022] FIG. 8 is a block diagram of a harmonic generator 800.

[023] FIGS. 9A, 9B, 9C, 9D, 9E and 9F show a set of graphs 900a-900f.

[024] FIG. 10 is a block diagram of a bass enhancement system 1000.

[025] FIG. 11 is a mobile device architecture 1100 for implementing the features and processes described herein, in accordance with one embodiment.

[026] FIG. 12 is a flowchart of an audio processing method 1200.

DETAILED DESCRIPTION

[027] Techniques related to bass enhancement are described here. In the following description, for purposes of explanation, numerous specific details are presented in order to provide a complete understanding of the present specification. It will be apparent, however, to one skilled in the art that the present specification, as defined by the claims, may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features. and concepts described in this document.

[028] In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is primarily for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step can begin. Such a situation will be specifically pointed out when it is not clear from the context.

[029] In this document, the terms “and”, “or” and “and/or” are used. Such terms should be read as having an inclusive meaning. For example, "A and B" can mean at least the following: "A and B", "at least A and B". As another example, "A or B" can mean at least the following: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and/or B" can mean at least the following: "A and B", "A or B". When an option or is intended, such a condition will be specifically observed (e.g., "A or B", "at most one of A and B").

[030] This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures can be implemented by a processor that is controlled by one or more computer programs.

[031] FIG. 1 is a block diagram of an audio processing system. The audio processing system 100 generally receives an input audio signal 102, processes the input audio signal 102 in accordance with the bass enhancement processes described herein, and generates an output audio signal 104. Audio processing system 100 includes a signal transform system 110, a bass enhancement system 120, an additional processing system 130 (optional), and an inverse signal transform system 140. The audio processing system 100 may include other components that (for brevity) are not discussed in detail. The components of the audio processing system 100 may be implemented by one or more computer programs that are executed by a processor.

[032] The signal transform system 110 receives the input audio signal 102, performs a signal transform process, and generates a transformed audio signal 112. The input audio signal 102 may be a time domain signal digital that includes a number of samples that correspond to audio (for example, sound in waveform pulse code modulation (PCM) format). The input audio signal 102 may have a sampling rate of 32 kHz, 44.1 kHz, 48 kHz, 192 kHz, etc. The input audio signal 102 may originate from a variety of formats, including the Advanced Television Systems Committee (ATSC) Digital Audio Compression Standard (AC-3, E-AC-3). As a specific example, input audio signal 102 may originate from a Dolby Digital Plus® signal with a sampling rate of 48 kHz.

[033] The signal transform system 110 can perform a variety of signal transform processes. In general, the signal transform process transforms the input audio signal 102 from a first signal domain to a second signal domain. For example, the first domain may be the time domain and the second domain of the signal may be the frequency domain, the quadrature mirrored frequency domain (QMF), the complex quadrature mirrored frequency domain (CQMF), the of hybrid complex quadrature mirror frequency (HCQMF), etc. The transform from the first signal domain to the second signal domain may also be referred to as "analysis", e.g. transform analysis, signal analysis, filter bank analysis, QMF analysis, CQMF analysis, HCQMF analysis , etc.

[034] In general, QMF domain information is generated by a filter whose frequency response is the mirror image around π/2 of that of another filter; together, these filters are known as a QMF pair. QMF theory also comprises filter banks with more channels than two (e.g., 64 channels); these can be referred to as M-channel QMF banks. QMF theory further teaches M-channel Pseudo QMF banks of the class referred to as modulated filter banks. In general, "CQMF" domain information results from a bank of complex-modulated discrete Fourier transform (DFT) filters applied to a time-domain signal. CQMF is a "complex" signal because it includes complex-valued signals, for example, signals that include an imaginary part in addition to the real part. In general, the “HCQMF” domain information corresponds to the CQMF domain information where the CQMF filter bank has been extended to a hybrid structure to achieve efficient non-uniform frequency resolution that better matches the frequency resolution of the human auditory system. In general, the term "hybrid" refers to a structure in which at least one frequency band is divided into subbands.

[035] According to a specific implementation of HCQMF, HCQMF information is generated in 77 frequency bands, where the lower CQMF bands are divided into subbands to obtain higher frequency resolution for the lower frequencies. According to a further specific implementation, the signal transform system 110 transforms each channel of the input audio signal 102 into 64 CQMF bands and further divides the 3 lowest bands into sub-bands as follows: the first band is divided into 8 sub-bands and the second and third bands are divided into 4 sub-bands. (This hybrid division of the lower bands into subbands is to improve the low frequency resolution of these bands.) The signal transform system 110 may include Nyquist filters to divide the bands into subbands. The 77 HCQMF bands then correspond to the 61 highest CQMF bands, plus the 16 subbands (8+4+4) of the 3 lowest CQMF bands. Subbands and bands can be numbered from 0 to 76, with the lowest frequency subband being number 0. The other subbands are then numbered from 1 to 15, and the remaining bands are numbered from 16 to 76. These 77 HCQMF bands can then be referred to as "hybrid bands" or "channels" along with their number, e.g. hybrid band 0, hybrid band 1, hybrid band 76, channel 0, channel 1, channel 76, etc. . Hybrid bands 0-15 may also be referred to as "sub-bands" along with their number, e.g. sub-band 0, sub-band 1, sub-band 15, etc. Hybrid bands 16-76 may also be referred to as "bands" along with their number, e.g. band 16, band 17, band 76, etc. Channels 1 and 3 may have passbands on the negative frequency axis, but generally the other channels do not.

[036] (It should be noted that the terms QMF, CQMF and HCQMF are presently used somewhat colloquially. Specifically, the terms QMF/CQMF may be used colloquially to refer to a bank of DFT filters that may include more than two bands. The term HCQMF may be used colloquially to refer to a non-uniform DFT filter bank that may include more than two bands)

[037] As a specific example, signal transform system 110 performs a HCQMF transform on input audio signal 102 to generate transformed audio signal 112 with 77 frequency bands. In this case, the signal domain of the transformed audio signal 112 may be referred to as the HCQMF domain or the hybrid domain, and the HCQMF transform may be referred to as HCQMF analysis.

[038] The bandwidth and sampling frequency of the bands will depend on the sampling frequency of the input audio signal 102. For example, when the input audio signal 102 has a sampling frequency of 48 kHz (corresponding to a maximum bandwidth of 24 kHz), the 77-band hybrid structure discussed above results in a sampling frequency of 750 Hz for all bands. The 61 bands with the highest frequencies have a pass bandwidth of 375 Hz; the 8 lowest frequency sub-bands have a pass bandwidth of 93.75 Hz; and the lower frequency sub-bands have a pass bandwidth of 187.5 Hz.

[039] The bass enhancement system 120 receives the transformed audio signal 112, performs bass enhancement, and generates an enhanced audio signal 122. In general, the bass enhancement system 120 generates harmonics for the transformed audio signal 112 so that the listener can psychoacoustically perceive the missing fundamental. More details of the bass enhancement system 120 are provided below (e.g., with reference to FIG. 2, etc.).

[040] Additional processing system 130 is optional. When present, the additional processing system 130 receives the enhanced audio signal 122, performs additional signal processing, and generates a processed audio signal 132. Alternatively, the additional processing system 130 may operate on the transformed audio signal 112 prior to processing. operation of the bass enhancement system 120, in which case the bass enhancement system 120 receives as its input the signal output of the additional processing system 130 (instead of receiving the output signal directly from the signal transform system 110 ). As another option, the additional processing system 130 may be multiple additional processing systems that operate before and after the bass enhancement system 120. The specific arrangement of the additional processing system 130 within the audio processing system 100 may vary from according to the specific types of additional processing that the additional processing system 130 performs.

[041] In general, the additional processing system 130 performs additional processing of the input audio signal 102 in the transform domain. This allows the bass enhancement system 120 to operate in combination with existing audio processing techniques that are implemented in the transform domain. Examples of additional processing include dialogue enhancement, intelligent equalization, volume leveling, spectral limiting, etc. Dialogue enhancement refers to the enhancement of speech signals (e.g., compared to sound effects) in order to improve speech intelligibility. Intelligent equalization refers to performing dynamic adjustment of audio tone, for example, to provide consistency of spectral balance (also known as "tone" or "timbre"). Volume leveling refers to increasing the volume of audibility and decreasing the volume of audibility, for example, to reduce the need for a listener to perform manual volume adjustment. Spectral limiting refers to limiting selected frequencies or frequency bands, for example, to limit lower frequencies that are difficult to output from small speakers.

[042] The inverse signal transform system 140 receives the enhanced audio signal 122 (or, optionally, the processed audio signal 132), performs an inverse transform, and generates the output audio signal 104. The inverse transform generally converts a signal from the second signal domain back to the first signal domain. In general, the inverse transform is an inverse of the signal transform process performed by the signal transform system 110. For example, when the signal transform system 110 performs a HCQMF transform, the inverse signal transform system 140 performs a inverse HCQMF transform. Transformation from the second signal domain back to the first signal domain may also be referred to as "synthesis", e.g., transform synthesis, signal synthesis, filterbank synthesis, etc.; and the inverse transform of HCQMF can be referred to as HCQMF synthesis.

[043] In this way, the output audio signal 104 corresponds to the input audio signal 102, with the addition of bass enhancement and/or additional signal enhancements. The output audio signal 104 can then be output from a speaker and perceived as sound by the listener.

[044] As discussed above and in more detail below, the bass enhancement system 120 is suitable for small to medium sized speakers. The processes implemented by the bass enhancement system 120 may be simpler than many existing bass enhancement methods; Compared to these existing methods, the bass enhancement system 120 has lower computational complexity and allows for short latency while maintaining audio quality. The 120 bass enhancement system is well suited for medium-sized speakers in, for example, TV sets or wireless speakers, and is also effective for bass enhancement of small transducers, for example, for telephones cell phones, laptops and tablets. The 120 bass enhancement system in one mode of operation not only adds harmonics to the mix, but also adds the original (dynamically changed) bass, that is, it can be operated to have an inherent bass boost.

[045] FIG. 2 is a block diagram of a bass enhancement system 200. The bass enhancement system 200 can be used as the bass enhancement system 120 (see FIG. 1). For brevity, the description of FIG. 2 focuses on a single signal processing path in order to describe the overall operation of the bass enhancement system 200; Additional signal processing paths may also be implemented in variations of the bass enhancement systems described herein (see, for example, FIG. 10). Additional signal processing pathways will also be briefly described here.

[046] The bass enhancement system 200 receives the transformed audio signal 112 (see FIG. 1). As discussed above, the transformed audio signal 112 is a hybrid complex transform domain signal (e.g., a HCQMF domain signal) having a number of bands (e.g., 77 hybrid bands, with the 3 lowest frequency bands divided in sub-bands). As a complex signal, the transformed audio signal 112 has complex values, for example, real values and imaginary values. Each subband can be processed in its own processing path, so the following description focuses on processing one subband (e.g., one of subbands 0, 2, 4, 6, etc.) . The bass enhancement system 200 includes an upsampler (optional) 202, a harmonic generator 204, a dynamics processor 206 (optional), a converter 208 (optional), a filter 212, a delay 214, and a mixer 216.

[047] Sampler 202 receives the transformed audio signal 112, upsamples it, and generates an upsampling signal 220. As an example, when the input audio signal 102 (see FIG. 1) has a sampling frequency of 48 kHz and the transformed audio signal 112 is processed into 64 bands, each band has a sampling frequency of 750 Hz. The sampler 202 can sample the selected sub-band of the transformed audio signal 112 by 2x, 3x, 4x, 5x , 6x, etc. A suitable amount of subsampling is 4x, for example, so that the subsampling signal 220 has a sampling frequency of 3 kHz when the selected subband of the transformed audio signal 112 has a sampling frequency of 750 Hz. The signal subsampling 220 is a complex transform domain signal. The subsampling signal 220 has a bandwidth that corresponds to the bandwidth of the selected subband of the transformed audio signal 112. As an example, when the selected subband 0 having a pass bandwidth of 93.75 Hz is inserted into the sampler, the sampled signal 220 also has a bandwidth of 93.75 Hz.

[048] Sampler 202 can be implemented by performing CQMF synthesis. As an example, to increase subband 0 from 750 Hz to 3000 Hz (4x boost), the sampler can implement 4-channel CQMF synthesis, with one input being subband 0 and the other 3 inputs being zero. (null). The synthesis is configured to maintain signal 220 as a complex-valued time domain signal.

[049] Sampler 202 is optional. In general, the sampler 202 provides additional headroom when generating the harmonics (see harmonic generator 204), to allow bandwidth extension without overlapping (also referred to as spectral folding). Sampler 202 may be omitted when processing one or more of the lower frequency subbands. For example, when processing only the lowest band (e.g., subband 0), sampler 202 can be omitted, because harmonics of up to (at least) 6th order can be generated without doubling. By processing the two lowest bands (e.g., subbands 0 and 2), sampler 202 can be omitted if only 2nd and 3rd order harmonics are generated. By processing the three lowest bands (e.g. subbands 0, 2 and 4), only 2nd order harmonics can be generated without overlapping. This is discussed in more detail with reference to harmonic generator 204.

[050] The harmonic generator 204 receives the sampled signal 220 (or the selected subband signal of the transformed audio signal 112 when the sampler 202 is omitted) and generates harmonics thereof to result in a signal 222. As mentioned with Reference to sampler 202, harmonic generator 204 extends the bandwidth of its input signal by generating the harmonics for signal 222. For example, when subrange 0 covers 0 to 93.75 Hz, the sampling frequency of 750 Hz may be sufficient to avoid overlapping of the generated harmonics. Likewise, when subband 2 covers 93.75 to 187.5 Hz, the sampling frequency of 750 Hz may be sufficient to avoid overlapping the generated harmonics. However, when subband 4 covers 187.5 to 281.25 Hz, the harmonics are approaching the Nyquist frequency of the original signal (with the 750 Hz sampling frequency), so up sampling is recommended for subbands 4 , 6, etc. Signal 222 is a complex transform domain signal. The signal 222 has a bandwidth that is greater than the bandwidth of the input to the harmonic generator 204, due to the addition of the harmonic frequencies. For example, when the sampled signal 220 has a bandwidth of 93.75 Hz, the signal 222 may have a bandwidth that exceeds 300 Hz.

[051] Harmonic generator 204 uses a non-linear process to generate harmonics. In general, a nonlinear process applies different gains to different components of the signal. Examples of the non-linear processes include multiplication, a feedback delay loop, rectification, etc., as detailed below with reference to FIGS. 3, 4, 5 and 8.

[052] Harmonic generator 204 can also perform audibility expansion when generating signal 222. As the sound pressure level for a fixed audibility range (in phon) is increasing with the frequency in the bass/mid range (e.g. , less than 800 Hz), harmonic generator 204 performs expansion in dynamics when generating signal 222. Examples of audibility expansion processes include dynamic compression and audibility correction. Further details of the audibility expansion are provided with reference to FIG. 6 below.

[053] Dynamic processor 206 receives signal 222, performs dynamic processing, and generates signal 224. Signal 224 is a complex transform domain signal. In general, the dynamics processor 206 implements dynamics processing by performing compression on the signal 222 in order to control the transient to tonal ratio of the signal 224. The dynamics processor 206 may implement an attack time that is relatively longer (e.g. example, between 4x to 12x longer, such as 8x longer) than the release time. For example, the attack time may be between 140 and 180 ms (e.g., 160 ms) and the release time may be between 15 and 25 ms (e.g., 20 ms). The dynamics processor 206 may implement decoupled soft peak detection using feed forward topology. The dynamics processor 206 can implement compression similar to the compression performed by the harmonic generator (described in more detail with reference to FIGS. 3, 4 and 5

[054] The dynamics processor 206 is optional. When dynamics processor 206 is omitted, converter 208 receives signal 222 instead of signal 224.

[055] Converter 208 receives signal 224 (or signal 222 when dynamics processor 206 is omitted), drops the imaginary part of signal 224, and generates signal 228. In general, dropping the imaginary part reduces computational complexity of subsequent analysis filter banks (e.g., filter 212), due to processing real-valued signals rather than complex-valued signals. As discussed above, signal 224 is a complex transform domain signal that has complex values, e.g., real values and imaginary values. The converter 208 may discard the imaginary part of the signal 224 by taking the real part of the complex-valued signal. Signal 228 is a real-valued transform domain signal.

[056] Converter 208 is optional and may be omitted in some embodiments of bass boost system 200. When sampler 202 is omitted, converter 208 must also be omitted, so that the imaginary part remains in the signal processing path for use by subsequent components.

[057] Filter 212 receives signal 228 (either signal 224 when converter 208 is omitted, or signal 222 when dynamics processor 206 and converter 208 are omitted), performs input filtering and generates signal 230 Signal 230 is a complex-valued transform domain signal. Filtering generally divides signal 228 into subbands as one of the inputs to mixer 216. The specifics of the filtering will depend on whether upsampling was performed or not (see upsampler 202).

[058] When sampler 202 is not present, filter 212 can be implemented by feeding the input signal (e.g., signal 228) into an 8-channel Nyquist filter bank to generate signal 230 that has hybrid subbands 0-7.

[059] When sampler 202 is present, filter 212 can be implemented by a bank of CQMF analysis filters and two or more Nyquist filters. The real part of the input signal (e.g., signal 228) is fed into the CQMF analysis filter bank; The CQMF analysis filter bank has an appropriate number of channels to generate signal 230 with 750 Hz sampling frequency subband signals. The appropriate number of channels depends on the sampling performed. For example, when a 4x upsampling is performed and therefore a 4-channel CQMF analysis bank is used in filter 212, the three lowest frequency CQMF subband signals are each fed into a Nyquist filter. corresponding (one generating hybrid sub-bands 0-7, one generating hybrid sub-bands 8-11 and one generating hybrid sub-bands 12-15). As another example, when 2x sampling is performed and therefore a 2-channel CQMF analysis bank is used in filter 212, the two CQMF subband signals are each fed into a corresponding Nyquist filter (one generating hybrid sub-bands 0-7 and one generating hybrid sub-bands 8-11). The remaining CQMF channels, if any, are provided to mixer 216 (with an appropriate delay corresponding to the delay of the Nyquist filters).

[060] Filter 212 can be implemented with filters similar to those used by signal transform system 110 (see FIG. 1). For example, a first Nyquist analysis filter with 8 channels can generate subbands 0-7, a second Nyquist analysis filter with 4 channels can generate subbands 8-11, and a third Nyquist analysis filter with 4 channels can generate sub-bands 12-15.

[061] Delay 214 receives the transformed audio signal 112, implements a delay period, and generates a signal 232. Signal 232 corresponds to a delayed version of the transformed audio signal 112 according to the delay period. Delay 214 can be implemented using a memory, a shift register, etc. The delay period corresponds to the processing time of the other components in the signal processing chain, for example, the sampler 202, the harmonic generator 204, the dynamics processor 206, the converter 208, the filter 212, etc. Because some of these other components are optional, the delay period decreases as more of the optional components are omitted. In one example, the delay period is 961 samples, of which 577 correspond to the upsampling and 384 correspond to the remaining components, for example, the Nyquist filters. As another example, the delay period is 384 samples when sampler 202 is omitted.

[062] Mixer 216 receives signal 230 and signal 232, performs mixing and generates enhanced audio signal 122 (see FIG. 1). The enhanced audio signal 122 is a transform domain signal. The mixer 216 mixes the signals on a per-band basis. For example, signal 230 and signal 232 may each have 77 hybrid bands (e.g., 8+4+4+61 HCQMF bands), and mixer 216 mixes subband 0 of signal 230 with subband 0 of signal 232, mixes sub-band 1 of signal 230 with sub-band 1 of signal 232, etc. The 216 mixer does not need to mix all bands; one or more of the bands of signal 232 may be passed when generating the enhanced audio signal 122. For example, the higher frequency bands (e.g., one or more of the hybrid bands 16-77) of signal 232 may be passed without mixture.

[063] More details of the bass enhancement system 200 are provided below. First, various options for the harmonic generator 204 are discussed, with reference to FIGS. 3-5.

[064] A A FIG. 3 is a block diagram of a harmonic generator 300. The harmonic generator 300 can be used as the harmonic generator 204 (see FIG. 2). In general, harmonic generator 300 generates each consecutive harmonic by multiplying (e.g., using direct signal multiplication) the input signal and the preceding harmonics.

[065] The harmonic generator 300 includes one or more multipliers 302 (two shown: 302a and 302b), two or more gain stages 304 (three shown: 304a, 304b and 304c), two or more compressors 306 (three shown: 306a, 306b and 306c) and two or more adders 308 (three shown: 308a, 308b and 308c). In general, each row of components in the harmonic generator 300 corresponds to one of the generated harmonics, so the number of rows (and corresponding number of components) can be adjusted to implement the desired number of harmonics. The first processing line includes the gain stage 304a, the compressor 306a, and the adder 308a. The second processing line includes the multiplier 302a, the gain stage 304b, the compressor 306b, and the adder 308b. The third processing line includes the multiplier 302b, the gain stage 304c, the compressor 306c, and the adder 308c. Additional lines can be added to generate additional harmonics, with each new line connected to the previous line in a similar way to what is shown in the figure.

[066] Harmonic generator 300 receives an input signal 320, also denoted as "x". Input signal 320 corresponds to sampled signal 220 (see FIG. 2) when sampler 202 is present, or to transformed audio signal 112 when sampler 202 is not present. The input signal 320 is a complex transform domain signal. For example, input signal 320 may correspond to a HCQMF band (e.g., hybrid subband 0, hybrid subband 2, hybrid subband 4, hybrid subband 6, etc.). Harmonic generator 300 generates signal 222 (see FIG. 2).

[067] Starting with the multipliers 302, the multiplier 302a receives the input signal 320, performs the multiplication of the input signal 320 with itself and generates a signal 322a, also denoted as “x2”. The multiplier 302b receives the input signal 320 and the signal 322a, performs the multiplication of the input signal 320 with the signal 322a and generates a signal 322b, also denoted as “x3”. Note that the output of a given multiplier is given as an input to the multiplier on the subsequent processing line: Signal 322a is given to multiplier 302b, signal 322b is given to the multiplier on the subsequent line (shown with a dotted line), etc. .

[068] Returning to the gain stages 304, the gain stage 304a receives the input signal 320, applies a gain g 1 and generates a signal 324a. Gain stage 304b receives signal 322a, applies a gain g 2 and generates a signal 324b. Gain stage 304c receives signal 322b, applies a gain g 3 and generates a signal 324c. The gains gi, g2, g3, etc. may be adjusted as desired, generally as a tuning exercise for each specific device that implements the harmonic generator 300. In general, the gi gain may be much less than the other gains (e.g., less than 50% of the other gains ). We set the gain gi to a small reduced value of what is referred to as the direct signal corresponding to the original bass harmonic, which is placed in small speakers that are physically unsuitable for reproducing any signal in the frequency range of the direct signal. If desired, the gain gi can be set to zero to eliminate the forward signal.

[069] Returning to compressors 306, compressor 306a receives signal 324a, performs dynamic compression and generates a signal 326a. The compressor 306b receives the signal 324b, performs dynamic compression and generates a signal 326b. The 306c compressor receives the 324c signal, performs dynamic compression and generates a 326c signal. Dynamic compression generally corresponds to an equation yr, where y corresponds to the input signal (for example, the 324a signal) and r is the compression ratio, where r is less than 1. The compression ratio r can differ for each harmonic ( e.g. each line). For example, the compression ratio ri for compressor 306a may differ from compression ratio r2 for compressor 306b, which may differ from compression ratio r3 for compressor 306c, etc. Compression ratios can be adjusted as tuning parameters based on the specific physical characteristics of the device implementing the harmonic generator 300. More details of the compressors 306 are provided below in the discussion of audibility expansion.

[070] Returning to adders 308, adder 308c receives signal 326c (and any adder output signal on any additional line), performs the addition and generates a signal 328b. The adder 308b receives the signal 326b and the signal 328b, performs addition and generates a signal 328a. Adder 308a receives signal 326a and signal 328a, performs addition, and generates signal 222 (see FIG. 2). Note that one of the inputs to a given adder is provided by the adder on the subsequent processing line: Adder 308c receives the output of the adder on the subsequent processing line (shown with a dotted line), adder 308b receives the output of adder 308c , adder 308a receives the output of adder 308b, etc.

[071] The harmonic generator 300 is processing complex valuable signals, for example, signals with very low contribution of negative frequencies. Thus, by generating harmonics by multiplying the complex-valued signal with itself, a much cleaner output is obtained than if the input signal is real-valued, for example, less intermodulation distortion results. In the complex case, for an input signal consisting of plural frequencies, only the desired terms plus the frequency sum terms are generated, but not the frequency difference terms, as would be the case in real-value processing. Difference terms are, although generally of low frequencies, more perceptually offensive than sum terms. Summation terms may actually be desirable, for example, when the input signal contains a harmonic series.

[072] FIG. 4 is a block diagram of a harmonic generator 400. The harmonic generator 400 can be used as the harmonic generator 204 (see FIG. 2). In general, harmonic generator 400 generates harmonics by applying a feedback delay loop to the input signal. The harmonic generator 400 includes a multiplier 402, a gain stage 404, a sum stage 406, a compressor 408, a delay stage 410, a gain stage 412, and a gain stage 414.

[073] The harmonic generator 400 receives an input signal 420. The input signal 420 corresponds to the sampled signal 220 (see FIG. 2) when the sampler 202 is present, or to the transformed audio signal 112 when the sampler 202 is not present. is present. The input signal 420 is a complex transform domain signal. For example, input signal 420 may correspond to a HCQMF band (e.g., hybrid subband 0, hybrid subband 2, hybrid subband 4, hybrid subband 6, etc.). Harmonic generator 400 generates signal 222 (see FIG. 2).

[074] The multiplier 402 receives the input signal 420, multiplies the input signal 420 with a signal 432, and generates a signal 422. The signal 432 may also be referred to as the feedback signal 432 and is discussed in more detail below with reference to gain stage 412.

[075] Gain stage 404 receives input signal 420, applies a gain a and generates a signal 424. The gain a can also be referred to as the mixing gain. The gain value a can be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonic generator 400.

[076] The addition stage 406 receives the signal 422 and the signal 424, performs the addition and generates a signal 426. The combination of the gain stage 404 and the addition stage 406, when added to the signal 422, is used to help to start the feedback loop (for example, when signal 432 is initially zero) and otherwise helps keep the feedback loop alive.

[077] Compressor 408 receives signal 426, performs dynamic compression and generates a signal 428. Dynamic compression generally corresponds to an equation yr, where y corresponds to the input signal (for example, signal 324a) and r is the rate of compression, where r is less than 1. The compression ratio can be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonic generator 400. More details of the compressor 408 are provided below in the discussion of expansion of audibility.

[078] Delay stage 410 receives signal 428, performs a delay operation, and generates a signal 430. Delay stage 410 can be implemented using a memory.

[079] Gain stage 412 receives signal 430, applies a gain g and generates signal 432. The gain g can also be referred to as the feedback gain. As discussed above in connection with multiplier 402, signal 432 is multiplied with input signal 420 to generate harmonics of theoretically undefined order.

[080] Gain stage 414 receives signal 428, applies a gain h, and generates signal 222 (see FIG. 2). The gain h can also be referred to as the output gain. The gain value h can be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonic generator 400.

[081] As with the harmonic generator 300, the harmonic generator 400 generates a direct signal corresponding to the original bass harmonic. The direct signal can be reduced, as desired, by adjusting the values of the gain a and the compression ratio r.

[082] As with the harmonic generator 300, the harmonic generator 400 is processing complex-valued signals, and by generating harmonics by multiplying the complex-valued signal with itself, a much cleaner output is obtained than if the signal input is of real value.

[083] FIG. 5 is a block diagram of a harmonic generator 500. The harmonic generator 500 can be used as the harmonic generator 204 (see FIG. 2). The harmonic generator 500 is similar to the harmonic generator 400 (see FIG. 4), but with the mix gain signal added after the compressor. The harmonic generator 500 includes a multiplier 502, a compressor 504, a gain stage 506, a sum stage 508, a delay stage 510, a gain stage 512, and a gain stage 514.

[084] The harmonic generator 500 receives an input signal 520. The input signal 520 corresponds to the sampled signal 220 (see FIG. 2) when the sampler 202 is present, or to the transformed audio signal 112 when the sampler 202 is not present. is present. The input signal 520 is a complex transform domain signal. For example, input signal 520 may correspond to a HCQMF band (e.g., hybrid subband 0, hybrid subband 2, hybrid subband 4, hybrid subband 6, etc.). Harmonic generator 500 generates signal 222 (see FIG. 2).

[085] The multiplier 502 receives the input signal 520, multiplies the input signal 520 with a signal 532, and generates a signal 522. The signal 532 may also be referred to as the feedback signal 532 and is discussed in more detail below with reference to gain stage 512.

[086] Compressor 504 receives signal 522, performs dynamic compression and generates signal 524. In general, the dynamic compressor corresponds to an equation yr, where y corresponds to an input signal (for example, signal 522) and r is the compression ratio, where r is less than 1. The compression ratio can be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonic generator 500. More details of the compressor 504 are provided below in the discussion about audibility expansion.

[087] Gain stage 506 receives input signal 520, applies a gain a and generates a signal 526. The gain a can also be referred to as the mixing gain. The gain value a can be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonic generator 500.

[088] The addition stage 508 receives the signal 524 and the signal 526, performs the addition and generates a signal 528. The combination of the gain stage 506 and the addition stage 508, when added to the signal 524, is used to help to initiate the feedback loop (for example, when signal 532 is initially zero) and otherwise helps keep the feedback loop alive.

[089] The delay stage 510 receives the signal 528, performing a delay operation, and generates a signal 530. The delay stage 510 can be implemented using a memory.

[090] Gain stage 512 receives signal 530, applies a gain g, and generates signal 532. The gain g can also be referred to as feedback gain. ||As discussed above in relation to multiplier 502, signal 532 is multiplied with input signal 520 to generate harmonics of theoretically undefined order.

[091] Gain stage 514 receives signal 524, applies a gain h and generates signal 222 (see FIG. 2). The gain h can also be referred to as the output gain. The gain value h can be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonic generator 500.

[092] Compared to the harmonic generator 300 (see FIG. 3) and the harmonic generator 400 (see FIG. 4), the harmonic generator 500 avoids the direct signal path by adding the input signal 520 later in the loop (e.g. as sign 526). In such an arrangement, input signal 520 passes through multiplier 502 (in contrast to adder 406 in FIG. 4) as part of generating signal 222, so that signal 222 contains no direct signal.

[093] As with the harmonic generator 300 and the harmonic generator 400, the harmonic generator 500 is processing complex-valued signals, and by generating harmonics by multiplying the complex-valued signal with itself, a much cleaner output is obtained than if the input signal is real-valued.

Expansion of audibility

[094] As discussed above, because the sound pressure level for a fixed audibility range (in phon) is increasing with the frequency in the bass/midrange range (e.g., less than 800 Hz), the harmonic generators (e.g., example, the harmonic generator 204 of FIG. 2, the harmonic generator 300 of FIG. 3, the harmonic generator 400 of FIG. 4, the harmonic generator 500 of FIG. coming to generate their output signals. Harmonic generators may use compressors (e.g., compressors 306 of FIG. 3, compressor 408 of FIG. 4, compressor 504 of FIG. 5, etc.) when performing audibility expansion. Examples of audibility expansion processes include dynamic compression and audibility correction.

Dynamic Compression

[095] Harmonic generators can generate harmonics of the nth order using an operation corresponding to Equation (1):

[096] In Equation (1), n is the order of the harmonic, y is the output signal, x is the input signal, ejnΦ is a complex exponential function, j is an imaginary number and Φ is the phase. The output signal is generated by multiplying the input signal by itself n times. Consequently, increasing n would increase the order of the generated harmonic. (The right-hand side of Equation (1) serves later in this document as an illustration of why dynamic expansion results in dynamic compression when the signals have been multiplied together.)

[097] FIG. 6 is a graph 600 showing equal audibility curves. In the 600 graph, the x-axis is frequency in Hz and the y-axis is sound pressure level (SPL) in dB. Graphic 600 includes 6 graphics 602a, 602b, 602c, 602d, 602e and 602f (collectively, graphics 602). Each of the graphs 602 corresponds to an audibility level in phon, which is a logarithmic measurement of the magnitude of the perceived sound. Each of the graphs 602 may also be referred to as an equal audibility curve. Chart 602a corresponds to perception threshold, chart 602b corresponds to 20 phon, chart 602c corresponds to 40 phon, chart 602d corresponds to 60 phon, chart 602e corresponds to 80 phon, and chart 602f corresponds to 100 phon.

[098] When generating harmonics through the operation described by Equation (1), the dynamics are expanded by a ratio of n. Given this information, the equal intensity graphs 602 suggest the relationship from Equation (2):

[099] In Equation (2), the term k (f,n) consists of a residue expansion rate that is related to the fundamental frequency f and the order of harmonics n. The expansion ratio of residue k (f,n) is typically in the range of 1.1 - 1.4, depending on the fundamental frequency f and the order of harmonics n. When harmonics are generated according to Equation (1), the related expansion ratio k (f,n) can be achieved by compressing the output of the harmonic generator by a factor k(f,n)/n. (As an aside, the terms expansion and compression can generally be used synonymously, with "compression" used when the ratio is less than 1 and "expansion" used when the ratio is greater than 1. Thus, the factor K(f n) /n may be referred to as "compression" due to the divisor n).

[0100] In graph 600, lines 610 and 612 illustrate an example of audibility expansion. Line 610 indicates an audibility range between 20 and 80 phon for a fundamental frequency of 50 Hz. Line 612 corresponds to the generation of a 4th order harmonic of 50 Hz from 400 Hz with the same audibility range. An arrow 614 from 610 to 612 indicates the generation of the 4th order harmonic. The dynamic SPL range of the fundamental frequency (line 610) is approximately 38 dB within the audibility range of 20 to 80 phon, and the dynamic SPL range of the 4th order harmonic (line 612) is approximately 50 dB for the same range of audibility. Therefore, when generating a 4th order harmonic from an 80 phon 50 Hz fundamental, the harmonic needs to be attenuated by approximately 20 dB. When the fundamental has an intensity of 20 phon, the harmonic needs to be attenuated by almost 40 dB, an increase in required attenuation of approximately 20 dB.

[0101] The SPL-to-phon expansion ratio, also referred to as the audibility expansion, can be approximated according to Equation (3):

[0102] In Equation (3), R (f) is the SPL-to-phon expansion ratio, which has an inverse relationship with the frequency f.

[0103] The residue expansion ratio k(f, n) is given by Equation (4):

[0104] In Equation (4), the residue expansion ratio k(f, n) corresponds to a ratio between the SPL-to-phon expansion ratio of the fundamental frequency f and the SPL-to-phon expansion ratio of the harmonic n • f which corresponds to a ratio between the natural logarithm of n (the harmonic order) and a natural logarithm of f (the fundamental frequency). In other words, the residue expansion ratio k(f, n) determines the factor required when generating the nth harmonic from a fundamental frequency at f (in Hz). Equations (3) and (4) are in good agreement with the equal intensity curves of FIG. 6 in the range of 20-80 phon and between 20 and 1000 Hz. By using the harmonic generator 400 (see FIG. 4) or the harmonic generator 500 (see FIG. 5), the required dynamic compression can be performed accurately sufficient using a simple compressor with a constant ratio (e.g., such as compressor 408 or compressor 504).

[0105] The compressor may apply dynamic compression using a first-order averaging filter to avoid distortion due to per-sample normalization. The first-order averaging filter can process a control signal s, which can be calculated according to Equation (5):

[0106] In Equation (5), m is the sample number, c is a compression gain, and a is a weight between the control signal value for the previous sample versus the compression gain value for the current sample. The weight α can also be referred to as an exponential smoothing factor and corresponds to the pole in the first-order low-pass system.

[0107] The weight α can be calculated using Equation (6):

[0108] In Equation (6), fs is the sampling frequency and T is a time constant.

[0109] The compression gain c can be calculated using Equation (7):

[0110] In Equation (7), a and b are polynomial coefficients that are applied to each order of magnitude of the sample m of the input signal x. Applying the compression gain c (or 25 the smoothed versions of Equation (5)) to a signal x as c • x (or s • x) corresponds to a rational approximation of the signal (x) • lxlr, which is the absolute value of the signal x subject to a compression ratio r multiplied by the signal function x.

[0111] FIG. 7 is a graph 700 showing various compression gains c. In graph 700, the x-axis is the input power (of input signal x) in dB and the y-axis is the compression gain c in dB. Several curves are shown, each curve corresponding to a value for the compression ratio r. Specifically, 9 values for r in the range 0.5 to 1.0 are given: 0.5, 0.6, 0.65, 0.7, 0.73, 0.77, 0.8, 0.9 and 1.0, with each value corresponding to one of the curves in graph 700 (e.g., the value for r of 0.5 corresponds to the top curve). Note that the gains indicated in FIG. 7 are not exact; it is merely an illustration of the general concept. Also notable from graph 700 is that the gain is limited to low input power and given by the ratio b(0)/a(0). This prevents excessive gain from being applied in circumstances such as transients occurring after periods of signal silence. (Rather, this gain in combination with the time constant in Equation (6) allows more energy to pass through the compressor during, for example, percussive onsets, contributing to the perception of "punch" in the bass signal.)

Audibility correction

[0112] An alternative approach to achieving audibility expansion is to apply normalization of the input signal in a first step, prior to harmonic generation, followed by a gain adjustment stage. This is called audibility correction.

[0113] FIG. 8 is a block diagram of a harmonic generator 800. The harmonic generator 800 generally performs audibility correction using normalization of input signals. Amplitude normalization theoretically prevents dynamic expansion of harmonics (by the ratio n, since n > 2d) when generated according to Equation (1).

[0114] The harmonic generator 800 includes two or more normalization stages 802 (two shown: 802a and 802b), two or more multipliers 804 (two shown: 804a and 804b), two or more audibility correction stages 806 (two shown: 806a and 806b), two or more adders 808 (two shown: 808a and 808b), and an adder 810. In general, each row of components in the harmonic generator 800 corresponds to one of the harmonics generated, so that the number of lines (and the corresponding number of components) can be adjusted to implement the desired number of harmonics. The first processing line includes the normalization stage 802a, the multiplier 804a, the audibility correction stage 806a, and the adder 808a. The second processing line includes the normalization stage 802b, the multiplier 804b, the audibility correction stage 806b, and the adder 808b. Additional lines can be added to generate additional harmonics, with each new line connected to the previous line in a similar way to what is shown in the figure.

[0115] The harmonic generator 800 receives an input signal 820. The input signal 820 corresponds to the sampled signal 220 (see FIG. 2) when the sampler 202 is present, or to the transformed audio signal 112 when the sampler 202 is not present. is present. The input signal 820 is a complex transform domain signal. For example, input signal 820 may correspond to a HCQMF band (e.g., hybrid subband 0, hybrid subband 2, hybrid subband 4, hybrid subband 6, etc.). Harmonic generator 800 generates signal 222 (see FIG. 2).

[0116] Starting with the normalization stages 802, the normalization stage 802a receives the input signal 820, performs normalization, and generates a signal 822a. The 802b normalization stage receives the input signal 820, performs normalization and generates an 822b signal. Similarly to Equation (5), each of the normalization stages 802 may perform normalization using a first-order smoothing filter to avoid distortion caused by sample-to-sample normalization. The normalization stages 802 may perform normalization in a manner described by Equation (8):

[0117] In Equation (8), x (m) is the current sample m of the normalized version of the input signal x, x (m — 1) is the previous sample of the normalized version of the input signal, α is a factor of smoothing ex(m) is given by Equation (9:

[0118] In Equation (9), x (m) corresponds to the ratio between the complex value of the current sample of the input signal and the magnitude (also referred to as the absolute value) of the current sample of the input signal. The smoothing factor α can be adjusted as desired to control the desired smoothing time and depend on the dynamics of the input signal. A smaller α is applied during attack events (e.g., when a rapid increase in signal energy occurs) than under steady-state or decreasing energy conditions in order to avoid signal clipping.

[0119] Alternatively, the harmonic generator may use a single normalization stage (e.g., 802a), with the output signal (e.g., 822a) provided as an input to each of the multipliers 804.

[0120] Returning to the multipliers 804, the multiplier 804a receives the input signal 820 and the signal 822a, multiplies these signals together and generates a signal 824a. The 804b multiplier receives the 822b signal and the 824a signal, multiplies these signals together and generates an 824b signal. Signal 824a corresponds to the second harmonic, signal 824b corresponds to the third harmonic, etc. Note that the output of a given multiplier is provided as an input to the multiplier on the subsequent processing line: Signal 824a is provided to multiplier 804b, signal 824b is provided to the multiplier on the subsequent line (shown with a dotted line), etc.

[0121] Returning to the audibility correction stages 806, the audibility correction stage 806a receives the signal 824a, performs the audibility correction, and generates the signal 826a. The 806b audibility correction stage receives the 824b signal, performs audibility correction, and generates the 826b signal. In general, the audibility correction stages 806 apply dynamic expansion and attenuation of the normalized energy of the generated harmonics, in line with the equal audibility curves of FIG. 6 in order to maintain the audibility compared to the fundamental. To adjust the audibility, a correction factor k is defined, where k is a function of the harmonic order n, the smoothed magnitude of the fundamental xhat (see Equation (8)) and the hybrid band index b. This correction factor k is applied according to Equation (10):

[0122] In Equation (10), h n(m) is the audibility corrected harmonic and h n(m) consists of the normalized harmonic, for each harmonic, respectively.

[0123] As discussed above, bass enhancement processes can be carried out in one or more hybrid bands (e.g., one or more of sub-bands 0, 2, 4, 6, 7, 9, etc.). Various harmonics, for example, 2nd, 3rd, and 4th order, are generated in each band. If we let the center frequency approach the fundamental frequency in each band, we can calculate the SPL-to-phon ratio using one parameter: the order or harmonics n. As an example, the first hybrid band (e.g., subband 0) has a center frequency of 46.875 Hz (e.g., approximately 47 Hz) and the corresponding values of the ELC curves in FIG. 6 are listed in table 1: TABLE 1

[0124] In TABLE 1, the value in parentheses is the difference between the NPS in relation to the fundamental. A function that represents the SPL difference of a harmonic and its fundamental can be calculated according to Equation (11):

[0125] In Equation (11), Kb,n is a gain value in dB, Ab is a minimum attenuation value, X is a smoothed fundamental input energy on a logarithmic scale, while βb,n is a scaling parameter harmonic order dependent n of the input energy. And βb,n can be calculated according to Equation (12):

[0126] The correction factor on a linear scale can be calculated according to Equation (13):

[0127] In Equations (12) and (13), Ab, εb and nb are all constants with a hybrid band basis and can be evaluated by an optimized fit with the ELC curves of FIG. 6. The parameters listed in TABLE 2 will result in adequate accuracy for the first six hybrid bands and the resulting audibility correction factors are visualized in FIG. 9. For bands 6, 7 and 9, the generated harmonics are in the frequency range of 700 to 2000 Hz, where the ELC curves are assumed to be flat. The audibility correction stages 806 may calculate audibility correction factors using segmental linear approximation to save computational complexity. TABLE2

[0128] FIGS. 9A, 9B, 9C, 9D, 9E and 9F show a set of graphs 900a-900f. In each graph, the x-axis is the magnitude of the normalized harmonic signal at the audibility correction stage (e.g., the input signal 824a at the audibility correction stage 806a, etc.) and the y-axis is the correction factor k . Graph 900a corresponds to hybrid band 0, graph 900b corresponds to hybrid band 2, graph 900c corresponds to hybrid band 4, graph 900d corresponds to hybrid band 6, graph 900e corresponds to hybrid band 7, and graph 900f corresponds to hybrid band 9. Lines for three harmonics (2nd, 3rd, and 4th order) are shown in each graph, but the lines are overlapping in graphs 900d, 900e, and 900f as the lines converge with the increasing number of hybrid bands. In general, the lines show the audibility correction factors k for the first 6 hybrid bands when using the hybrid band-based constants listed in TABLE 2.

[0129] Returning to FIG. 8 and adders 808, adder 808b receives signal 826b (and any signal received from the subsequent processing line, shown with a dotted line), performs the addition, and generates a signal 828b. The adder 808b receives the signal 826a and the signal 828b, performs addition and generates a signal 828a. Note that one of the inputs to a given adder is provided by the adder on the subsequent processing line: the adder 808b receives the output of the adder on the subsequent processing line (shown with a dotted line), the adder 808a receives the output of the adder 808b , etc.

[0130] Adder 810 receives input signal 820 and signal 828a, performs addition and generates signal 222 (see FIG. 2).

Hybrid multi-band processing

[0131] Although the description for the bass enhancement system 200 (see FIG. 2) focuses on processing a single hybrid band, similar processing can be performed on multiple hybrid bands. For example, the bass enhancement system 120 (see FIG. 1) may be realized in four hybrid bands (e.g., subbands 0, 2, 4, and 6), six hybrid bands (e.g., subbands 0 , 2, 4, 6, 7 and 9), etc. Various harmonics (e.g. 2°, 3°, 4°, etc.) are generated in each band.

[0132] FIG. 10 is a block diagram of a bass boost system 1000. The bass boost system 1000 can be used as the bass boost system 120 (see FIG. 1). The bass enhancement system 1000 is similar to the bass enhancement system 200 (see FIG. 2), with similar components having similar names and reference numbers, and the addition of multiple explicit processing paths. Each processing path corresponds to the processing of a hybrid subband signal. As a specific example, four processing paths are shown (e.g., to process hybrid subbands 0, 2, 4, and 6). The number of processing paths can be increased or decreased as desired. For example, six processing paths can be used to process hybrid subbands 0, 2, 4, 6, 7, and 9.

[0133] Bass enhancement system 1000 receives the transformed audio signal 112 (see FIG. 1). As discussed above, the transformed audio signal 112 is a hybrid complex transform domain signal with hybrid bands. Four of the hybrid bands of the transformed audio signal 112 are shown as the inputs to the bass boost system 1000: subband 0 (labeled 1002a), subband 2 (1002b), subband 4 (1002c), and subband 4 (1002c). -band 6 (1002d). Each subband corresponds to one of the processing paths. The bass enhancement system 1000 includes subsamplers 1010 (four shown: 1010a, 1010b, 1010c, and 1010d), harmonic generators 1012 (four shown: 1012a, 1012b, 1012c, and 1012d), an adder 1014, a dynamics processor 1016 (optional ), a converter 1018 (optional), a filter 1022, a delay 1024 and a mixer 1026.

[0134] The upsampler 1010a receives the signal 1002a, performs the upsampling, and generates an upsampling signal 1030a. The upsampler 1010b receives the signal 1002b, performs the upsampling, and generates an upsampling signal 1030b. The upsampler 1010c receives the signal 1002c, performs the upsampling, and generates an upsampling signal 1030c. The upsampler 1010d receives the signal 1002d, performs upsampling, and generates an upsampling signal 1030d. Signals 1030a, 1030b, 1030c and 1030d are complex transform domain signals. Upstream samplers 1010 are otherwise similar to those described above in connection with upstream sampler 202 (see FIG. 2).

[0135] The harmonic generator 1012a receives the sampled signal 1030a and generates harmonics thereof to result in a signal 1032a. The harmonic generator 1012b receives the sampled signal 1030b and generates harmonics thereof to result in a signal 1032b. The harmonic generator 1012c receives the sampled signal 1030c and generates harmonics thereof to result in a signal 1032c. The harmonic generator 1012d receives the sampled signal 1030d and generates harmonics thereof to result in a signal 1032d. Signals 1032a, 1032b, 1032c and 1032d are complex transform domain signals. Harmonic generators 1012 are otherwise similar to harmonic generator 204 (see FIG. 2). For example, one or more of the harmonic generators 1012 may be implemented using the harmonic generator 300 (see FIG. 3), the harmonic generator 400 (see FIG. 4), the harmonic generator 500 (see FIG. 5) , the harmonic generator 800 (see FIG. 8), etc.

[0136] Adder 1014 receives signals 1032a, 1032b, 1032c and 1032d, performs addition and generates a signal 1034. Signal 1034 is a complex transform domain signal.

[0137] The dynamics processor 1016 receives the signal 1034, performs dynamics processing, and generates a signal 1036. The signal 1036 is a complex transform domain signal. Dynamic processor 1016 is otherwise similar to dynamic processor 206 (see FIG. 2). The 1016 dynamics processor is optional. When the dynamics processor 1016 is omitted, the converter 1018 receives the signal 1034 instead of the signal 1036.

[0138] Converter 1018 receives signal 1036 (or signal 1034 when dynamics processor 1016 is omitted), drops the imaginary part of signal 1036, and generates signal 1040. Signal 1040 is a transform domain signal. Converter 1018 is otherwise similar to converter 208 (see FIG. 2), including being optional.

[0139] Filter 1022 receives signal 1040 (either signal 1036 when converter 1018 is omitted, or signal 1034 when dynamic processor 1016 and converter 1018 are omitted), performs filtering, and generates signal 1042. The signal 1042 is a transform domain signal. Filter 1022 is otherwise similar to filter 212 (see FIG. 2).

[0140] Delay 1024 receives signal 1042, implements a delay period, and generates a signal 1044. Signal 1044 corresponds to a delayed version of the transformed audio signal 112 according to the delay period. Delay 1024 may be implemented using a memory, a shift register, etc. The delay period corresponds to the processing time of the other components in the signal processing chain; Because some of these other components are optional, the delay period decreases when optional components are omitted. Delay 1024 is otherwise similar to delay 214 (see FIG. 2).

[0141] Mixer 1026 receives signal 1042 and signal 1044, performs mixing and generates enhanced audio signal 122 (see FIG. 1). Mixer 1026 is otherwise similar to mixer 216 (see FIG. 2).

[0142] FIG. 11 is a mobile device architecture 1100 for implementing the features and processes described herein, in accordance with one embodiment. The 1100 architecture can be implemented in any electronic device, including, but not limited to: a desktop computer, consumer audio/visual (AV) equipment, radio broadcast equipment, mobile devices (e.g., smartphone, tablet, laptop, wearable device), etc. In the example embodiment shown, architecture 1100 is for a portable computer and includes processor(s) 1101, peripheral interface 1102, audio subsystem 1103, speakers 1104, microphone 1105, sensors 1106 (e.g., accelerometers, gyroscopes, barometer , magnetometer, camera), location processor 1107 (e.g., GNSS receiver), wireless communications subsystems 1108 (e.g., Wi-Fi, Bluetooth, cellular), and I/O subsystem(s) 1109, which includes the touch controller 1110 and other input controllers 1111, touch surface 1112 and other input/control devices 1113. Other architectures with more or fewer components may also be used to implement the disclosed embodiments.

[0143] Memory interface 114 is coupled to processors 1101, peripheral interface 1102, and memory 1115 (e.g., flash, RAM, ROM). Memory 1115 stores computer program instructions and data, including, but not limited to: operating system instructions 1116, communication instructions 1117, GUI instructions 1118, sensor processing instructions 1119, telephone instructions 1120, electronic messages 1121, web navigation instructions 1122, audio processing instructions 1123, GNSS/navigation instructions 1124, and applications/data 1125. The audio processing instructions 1123 include instructions for performing the audio processing described herein.

[0144] FIG. 12 is a flowchart of an audio processing method 1200. Method 1200 may be performed by a device (e.g., a laptop computer, a mobile phone, etc.) with the components of architecture 1100 of FIG. 11, to implement the functionality of the audio processing system 100 (see FIG. 1), the bass enhancement system 200 (see FIG. 2), the bass enhancement system 1000 (see FIG. 10), etc. , for example, running one or more computer programs. In general, method 1200 performs audio signal processing in a complex-valued subband domain (e.g., the HCQMF domain).

[0145] In step 1202, a first transform domain signal is received. The first transform domain signal is a hybrid complex transform domain signal having a number of bands. At least one of the bands has a series of sub-bands. The first transform domain signal has a first plurality of harmonics. For example, the bass enhancement system 200 (see FIG. 2) may receive the transformed audio signal 112. The first transform domain signal may have 77 hybrid bands numbered 0-76, where bands 0-15 are sub -bands that result from the division of one or several larger bands. The first transform domain signal may be a CQMF domain signal. The first transform domain signal may be a HCQMF signal generated by dividing (e.g., using Nyquist filter banks) a subset of the channels of a CQMF domain signal into subbands to increase the frequency resolution to the range of lower frequency.

[0146] In step 1204, a second transform domain signal is generated based on the first transform domain signal. The second transform domain signal is generated by generating harmonics to the first transform domain signal according to a nonlinear process. The second transform domain signal has a second plurality of harmonics that differs from the first plurality of harmonics and the second transform domain signal is a complex valued signal having an imaginary part. The second transform domain signal is further generated by performing audibility expansion in the second plurality of harmonics. For example, the harmonic generator 204 (see FIG. 2), the harmonic generator 300 (see FIG. 3), the harmonic generator 400 (see FIG. 4), the harmonic generator 500 (see FIG. 5) , the harmonic generator 800 (see FIG. 8), etc. may generate the second transform domain signal (e.g., signal 222) based on the first transform domain signal (e.g., signal 220, etc.).

[0147] In step 1206, a third transform domain signal is generated by filtering the second transform domain signal. The third transform domain signal has a number of bands and at least one of the bands has a number of subbands. For example, filter 212 (see FIG. 2) may filter signal 228 (or signal 226) to generate signal 230. As another example, filter 1022 (see FIG. 10) may filter signal 1040 to generate signal signal 1042. The third transform domain signal may have 77 hybrid bands numbered 0-76, where bands 0-15 are sub-bands resulting from the division of one or more larger bands. The third transform domain signal may be an HCQMF domain signal.

[0148] In step 1208, a fourth transform domain signal is generated by mixing the third transform domain signal with a delayed version of the first transform domain signal. A given subband of the third transform domain signal is mixed with a corresponding subband of the delayed version of the first transform domain signal. For example, mixer 216 (see FIG. 2) may mix signal 230 with delayed signal 232. As another example, mixer 1026 (see FIG. 10) may mix signal 1042 with delayed signal 1044. input may have 77 hybrid bands numbered 0 to 76, where a given band of one input signal (e.g., band 0) is mixed with the corresponding band of the other input signal (e.g., band 0).

[0149] Method 1200 may include additional steps corresponding to the other functionalities of the bass enhancement system 200, the bass enhancement system 1000, etc., as described herein. For example, the fourth transform domain signal may be output by a speaker, such as speakers 1104 (see FIG. 11). As another example, transform domain signals may be sampled (e.g., using sampler 202, samplers 1010) before generating harmonics at 1204. As another example, dynamic processing may be applied to transform domain signals. , for example, using the dynamics processor 206 or the dynamics processor 1016. As another example, generating the harmonics may include performing multiplication, using a feedback delay loop, etc. As another example, the second transform domain signal may be a number of second transform domain signals, each of which corresponds to a hybrid band of the first transform domain signal. As another example, the imaginary part of the second transform domain signal may be discarded before generating the third transform domain signal.

IMPLEMENTATION DETAILS

[0150] An embodiment may be implemented in hardware, executable modules stored on a computer-readable medium, or a combination of both (e.g., programmable logic arrays). Unless otherwise specified, the steps performed by embodiments need not be inherently related to any particular computer or other device, although they may be in certain embodiments. In particular, various general-purpose machines may be used with programs written in accordance with the teachings of this document, or it may be more convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the necessary steps of the method. Thus, embodiments may be implemented in one or more computer programs executing on one or more programmable computer systems, each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory, and/or or storage elements), at least one input device or port and at least one output device or port. Program code is applied to input data to perform the functions described above and generate output information. Output information is applied to one or more output devices in a known manner.

[0151] Each of these computer programs is preferably stored on a storage media or device (e.g., ROM or magnetic floppy disk) readable by a general or special purpose programmable computer, to configure and operate the computer when the media or device storage is read by the computer to perform the procedures described in this document. The system may also be considered implemented as a computer-readable storage medium, configured with a computer program, wherein the storage medium so configured causes a computer to operate in a specific, predefined manner to perform the functions described in this document. . (The software itself and intangible or transient signs are excluded to the extent that they are non-patentable subject matter.)

[0152] Aspects of the systems described herein may be implemented in a computer-based sound processing network environment suitable for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks comprising any desired number of individual machines, including one or more routers (not shown) that serve to buffer and forward data transmitted between the computers. Such a network can be built on several different network protocols and can be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0153] One or more of the components, blocks, processes, or other functional components may be implemented through a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various logic and/or functions disclosed herein may be enabled using any number of combinations of hardware, firmware, and/or as data and/or instructions embedded in various machine-readable or computer-readable media, in terms of its behavior, record transfer, logical component and/or other characteristics. Computer-readable media in which such formatted data and/or instructions may be embodied include, but are not limited to, physical (non-transitory), non-volatile storage media in various forms, such as optical, magnetic or semiconductor storage media.

[0154] The above description illustrates various embodiments of the present specification, along with examples of how aspects of the present specification can be implemented. The above examples and embodiments should not be considered the only embodiments and are presented to illustrate the flexibility and advantages of the present specification, as defined by the following claims. Based on the above specification and the following claims, other arrangements, embodiments, implementations and equivalents will be apparent to those skilled in the art and may be employed without departing from the spirit and scope of the specification as defined by the claims.

Claims (17)

1. A computer-implemented audio processing method (1200), comprising: receiving (1202) a first transform domain signal (112), wherein the first transform domain signal (112) consists of a hybrid complex transform having a plurality of bands, wherein at least one of the plurality of bands has a plurality of subbands, wherein the first transform domain signal (112) has a first plurality of harmonics; the method CHARACTERIZED by the fact that it further comprises: generating an upsampled transform domain signal (220) by upsampling the first transform domain signal (112), wherein the upsampled signal (220) is a complex valued time domain signal; generating (1204) a second transform domain signal (222) based on the first upsampled transform domain signal (220) by: generating a second plurality of harmonics for the first upsampled transform domain signal (220) of according to a non-linear process, in which the second transform domain signal (222) has the second plurality of harmonics that differs from the first plurality of harmonics, and performing loudness expansion on the second plurality of harmonics, in which the second transform domain signal (222) is a complex-valued signal having an imaginary part; filtering the second transform domain signal (222) to divide the second transform domain signal (222) into a plurality of subbands and generating (1206) a third transform domain signal (230, 1042), wherein the third transform domain signal (230, 1042) has a plurality of bands, wherein at least one of the plurality of bands has the plurality of subbands; and generating (1208) a fourth transform domain signal (122) by mixing the third transform domain signal (230, 1042) with a delayed version (232, 1044) of the first transform domain signal (112), wherein a given subband of the third transform domain signal (230, 1042) is mixed with a corresponding subband of the delayed version (232, 1044) of the first transform domain signal (112). 2. Method, according to claim 1, CHARACTERIZED by the fact that the generation of the upsampled transform domain signal (220) is carried out according to complex quadrature mirror filtering synthesis. 3. Method, according to claim 1 or 2, CHARACTERIZED by the fact that it further comprises: performing dynamic processing on the second transform domain signal (222), before generating the third transform domain signal (230, 1042) from the second transform domain signal (222). 4. Method according to any one of claims 1 to 3, CHARACTERIZED by the fact that the plurality of bands of the first transform domain signal (112) has a first band, a second band and a third band, wherein the The first band is divided into 8 sub-bands, the second band is divided into 4 sub-bands, and the third band is divided into 4 sub-bands. 5. Method according to any one of claims 1 to 4, CHARACTERIZED by the fact that the first transform domain signal (112) has 64 bands, wherein a first band is divided into 8 sub-bands, wherein a second band is divided into 4 sub-bands, and a third band is divided into 4 sub-bands. 6. Method according to any one of claims 1 to 5, CHARACTERIZED by the fact that the first transform domain signal (112) has a bandwidth of 24 kHz, wherein the first transform domain signal (112 ) has 64 bands, and the passband bandwidth of each band is 375 Hz. 7. Method according to any one of claims 1 to 6, CHARACTERIZED by the fact that the non-linear process includes multiplication of the first transform domain signal (112). 8. Method according to any one of claims 1 to 7, CHARACTERIZED by the fact that the nonlinear process includes a feedback delay loop applied to the first transform domain signal (112). 9. Method according to any one of claims 1 to 8, CHARACTERIZED by the fact that generating (1204) the second transform domain signal (222) comprises: generating the second transform domain signal (222) based in a subband of the plurality of subbands of the first transform domain signal (112), wherein the one subband of the plurality of subbands is smaller than all of the subbands of the plurality of subbands of the first transform domain signal (112). 10. Method according to any one of claims 1 to 8, CHARACTERIZED by the fact that generating (1204) the second transform domain signal (222) comprises: generating a plurality of second transform domain signals (222) based on two or more of the plurality of subbands of the first transform domain signal (112), wherein the two or more subbands of the plurality of subbands are smaller than all of the subbands of the plurality of subbands of the first transform domain signal (112), and wherein each subband of the plurality of second transform domain signals (222) corresponds to a subband of the two or more subbands of the plurality of sub-bands; and generating the second transform domain signal (222) by summing the plurality of second transform domain signals (222). 11. Method, according to any one of claims 1 to 10, CHARACTERIZED by the fact that it further comprises: emitting, through a speaker (1104), sound corresponding to the fourth transform domain signal (122). 12. Method according to any one of claims 1 to 11, CHARACTERIZED by the fact that the first transform domain signal (112) is in a first signal domain, the method further comprising: receiving an input signal in a second signal domain; generating the first transform domain signal (112) by converting the input signal from the second signal domain to the first signal domain; and generating an output signal by converting the fourth transform domain signal (122) from the first signal domain to the second signal domain. 13. Method according to claim 12, CHARACTERIZED by the fact that the second transform domain is a time domain, wherein the first signal domain is a hybrid complex quadrature mirror filter (HCQMF) signal domain ; wherein generating the first transform domain signal (112) comprises generating the first transform domain signal (112) by performing HCQMF analysis on the input signal; and wherein generating the output signal comprises generating the output signal by performing HCQMF synthesis on the fourth transform domain signal (122). 14. Method, according to any one of claims 1 to 13, CHARACTERIZED by the fact that it further comprises: discarding the imaginary part of the second transform domain signal (222), before generating the third transform domain signal (230 , 1042). 15. Non-transitory computer-readable medium CHARACTERIZED by the fact that it stores a set of instructions that, when executed by a processor, controls an apparatus to perform processing, including the method as defined in any one of claims 1 to 14. 16. Audio processing apparatus (1100), comprising: a processor (1101); wherein the processor (1101) is configured to control the apparatus (1100) to receive a first transform domain signal (112), wherein the first transform domain signal (112) is a hybrid complex transform domain signal having a plurality of complex values and a plurality of bands, wherein at least one band of the plurality of bands has a plurality of subbands, wherein the first transform domain signal (112) has a first plurality of harmonics; CHARACTERIZED by the fact that the processor (1101) is configured to control the apparatus (1100) to: generate an upsampled transform domain signal (220) by upsampling the first transform domain signal (112), wherein the up-sampled signal (220) consists of a complex-valued time domain signal; and generating a second transform domain signal (222) based on the first upsampled transform domain signal (220) by: generating a second plurality of harmonics for the first upsampled transform domain signal (220) in accordance with a non-linear process, in which the second transform domain signal (222) has the second plurality of harmonics that differs from the first plurality of harmonics, and performing loudness expansion on the second plurality of harmonics, in which the second transform domain signal (222) is a complex valued signal having an imaginary part; wherein the processor (1101) is configured to control the apparatus (1100) to filter the second transform domain signal (222) to divide the second transform domain signal (222) into a plurality of subbands and generate a third transform domain signal (230, 1042), wherein the third transform domain signal (230, 1042) has a plurality of bands, wherein at least one band of the plurality of bands has a plurality of subbands; wherein the processor (1101) is configured to control the apparatus (1100) to generate a fourth transform domain signal (122) by mixing the third transform domain signal (230, 1042) with a delayed version (232, 1044) of the first transform domain signal (112), wherein a given subband of the third transform domain signal (230, 1042) is mixed with a corresponding subband of the delayed version (232, 1044) of the first transform domain signal (230, 1042) transform domain (112). 17. Apparatus, according to claim 16, CHARACTERIZED by the fact that it further comprises: a speaker (1104) which is configured to output the fourth transform domain signal (122) as sound.