JP6231762B2 - Receiving apparatus and program - Google Patents

Description

  The present invention relates to a receiving apparatus and a program, and more particularly to a receiving apparatus and a program capable of receiving both broadcast information and information transmitted in Internet communication.

  There are various demands for sound of broadcast programs, such as enhancement of multilingual broadcasting and explanation broadcasting, and setting of dialog sound level that is easy to hear. In particular, there is a great demand for easy-to-listen audio, and broadcasters have begun efforts to realize easy-to-listen program audio, for example, by introducing a loudness meter in program production. However, even if a program is produced so that it is easy to hear the sound, the ease of hearing the sound changes depending on the environment on the receiving side of the broadcast signal and the listener (listener's hearing ability). .

  Therefore, although it is difficult to extract only the desired audio from the program audio already produced on the broadcasting station side, it is desirable that the level of the dialog audio can be adjusted on the receiving device side in order to pursue ease of listening. It is. Conventionally, a method has been proposed in which a receiving apparatus uses a correlation between channels to control a volume difference between dialog sound (for example, narration sound) and other sound (for example, background sound or sound effect). (For example, refer to Patent Document 1). In the method described in Patent Document 1, the receiving apparatus first extracts a dialog voice signal that is an in-phase component with high correlation from two-channel audio signals included in a broadcast signal, and amplifies the dialog voice signal.・ Apply processing such as attenuation and filtering. Then, the reception device adds the processed dialog audio signal to the 2-channel audio signal from which the in-phase component value is extracted. As a result, various processes are performed on the dialog voice signal in the voice signal.

Japanese Patent No. 4970174

  However, not only the correlation of the dialog voice signal is high, but in the conventional method described in Patent Document 1, for example, a mono signal such as a background sound is considered to have a high correlation and is extracted together with the dialog voice signal. . In this case, the receiving apparatus cannot amplify or attenuate only the dialog voice signal.

  Further, the conventional method is merely intended to emphasize or weaken the dialog sound, and does not allow the dialog sound to be replaced on the receiving device side. That is, since the conventional receiving apparatus does not receive anything other than the broadcast signal, another signal other than the broadcast signal cannot be combined with the broadcast signal that has been processed for the dialog sound. Therefore, in the conventional method, in order to replace the original dialog sound (for example, Japanese) in the broadcast signal with another dialog sound (for example, English), the original dialog sound is deleted, that is, the level of the original dialog sound. There is no assumption about zero. Even if the dialog voice is deleted by the conventional method, for the above reason, it is difficult to delete only the dialog voice by using the correlation value, and the effect is insufficient.

  Accordingly, an object of the present invention made in view of the above problems is to provide a receiving apparatus and a program capable of adjusting the level of only dialog sound included in a received broadcast signal.

In order to solve the above problems, a receiving apparatus according to the present invention provides:
A first receiver for receiving broadcast program audio;
A second receiver for receiving the dialog voice of the program voice by Internet communication;
A determination unit that determines the validity of each frame based on the program sound and the dialog sound;
An estimation unit configured to estimate a gain amount of a dialog sound included in the program sound based on the program sound and the dialog sound of the frame determined to be valid by the determination unit;
An adjusting unit for adjusting the dialog sound based on the gain amount;
A synthesizing unit that synthesizes the dialog audio adjusted by the adjusting unit and the program audio;
Is provided.

The receiving device according to the present invention is
The determination unit determines the validity of each frame based on the frame energy of the dialog sound and the similarity between the program sound and the dialog sound.

The receiving device according to the present invention is
The determination unit divides a band of the program sound and the dialog sound into a plurality of bands, and determines that a frame having similarity in at least one band is valid.

The receiving device according to the present invention is
The estimation unit sets the gain amount of a frame that is not determined to be valid to be the same as the gain amount of the previous frame.

The receiving device according to the present invention is
The adjustment unit adjusts the dialog sound by dephasing the dialog sound and multiplying the gain by the gain amount.

Moreover, the receiving device according to the present invention includes:
A second synthesizing unit that further synthesizes the result synthesized by the synthesizing unit and the dialog sound different from the dialog sound among the dialog sounds received by the second receiving unit .

The program according to the present invention is
In the computer of the receiving device,
Obtaining broadcast program audio; and
Obtaining a dialog sound of the program sound by Internet communication;
Determining the validity of a frame based on the program audio and the dialog audio;
Estimating a gain amount of the dialog sound included in the program sound based on the program sound and the dialog sound of the frame determined to be valid;
Adjusting the dialog sound based on the gain amount;
And synthesizing the dialog sound and the program sound adjusted in the adjusting step .

  ADVANTAGE OF THE INVENTION According to this invention, the receiver and program which can adjust the level of only the dialog audio | voice contained in the received broadcast signal can be provided.

FIG. 1 is a functional block diagram showing a schematic configuration of a receiving apparatus according to the first embodiment of the present invention. FIG. 2 is a flowchart showing processing of the receiving apparatus according to the first embodiment of the present invention. FIG. 3 is a functional block diagram showing a schematic configuration of a receiving apparatus according to the second embodiment of the present invention. FIG. 4 is a flowchart showing processing of the receiving apparatus according to the second embodiment of the present invention.

  Hereinafter, a receiving apparatus according to an embodiment of the present invention will be described with reference to the drawings. The receiving device can receive broadcast signals, and is a broadcast receiving terminal such as a television receiver, for example. The receiving apparatus includes a computer that performs calculation processing described below. The receiving device is not limited to the presence or absence of a display unit (display) that displays information on the received broadcast signal, as long as it can receive the broadcast signal. For this reason, the receiver includes a recorder equipped with a tuner, a single tuner, and the like.

  Broadcasting refers to transmission of a broadcast signal from a broadcasting station to the public (unspecified majority), for example, television broadcasting. The broadcast signal is a combination of two signals of a program video (program video) and audio (program audio) provided by a broadcasting station. The program audio includes dialog voices such as voice narrations, announcer comments, vocals, and other sounds (background sounds, sound effects, etc.). In addition, the level of the dialog sound included in the program sound is changed with time by being subjected to a level change process in the process of producing the program.

  And this receiving apparatus receives not only a broadcast signal but the communication signal via the internet from a broadcast station. For this reason, the receiving device and the broadcasting station are wired or wirelessly connected. A receiving apparatus that receives both a broadcast signal and a communication signal is, for example, a receiving apparatus in Hybridcast (registered trademark), which is a basic system of a broadcast communication cooperation service.

(First embodiment)
FIG. 1 is a functional block diagram showing a schematic configuration of a receiving apparatus 1 according to the first embodiment of the present invention. The receiving device 1 of the present embodiment includes a first receiving unit 3, a second receiving unit 5, a determining unit 7, an estimating unit 9, an adjusting unit 11, and a combining unit 13. The first receiving unit 3 is connected to the determining unit 7 and the combining unit 13, and the second receiving unit 5 is connected to the determining unit 7 and the adjusting unit 11. Although not shown, the receiving device 1 includes a control unit that executes a program describing processing contents for realizing each function, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and the like. it can. In addition, the receiving device 1 stores a program describing various processing contents for realizing each function and various types of information, and a volatile storage such as a RAM (Random Access Memory) that can also function as a buffer. A non-volatile storage medium such as a medium or ROM (Read Only Memory) can be provided.

  The 1st receiving part 3 receives the broadcast signal broadcast by the broadcasting station, and isolate | separates the program video and program audio | voice which comprise a broadcast signal. Then, the first receiving unit 3 sends the program audio to the determining unit 7 and the synthesizing unit 13 for each frame.

  The second receiving unit 5 receives a communication signal from the broadcasting station via the Internet. In this embodiment, the communication signal is a dialog sound before the level change process. That is, the dialog sound included in the program sound via the first receiving unit 3 and the dialog sound via the second receiving unit 5 have different levels, and the level difference changes with time. To do. The dialog sound included in the program sound acquired from the first receiving unit 3 and the dialog sound acquired from the second receiving unit 5 are synchronized on the time axis except for the level. The second reception unit 5 sends the received dialog voice to the determination unit 7 and the adjustment unit 11 for each frame.

  The determination unit 7 has two inputs for program audio and dialog audio, and is connected to the first receiving unit 3 and the second receiving unit 5, respectively. The determination unit 7 has two outputs for program sound and dialog sound, and is connected to the estimation unit 9.

  A process for the determination unit 7 to determine the validity of each frame based on the program sound acquired from the first reception unit 3 and the dialog sound acquired from the second reception unit 5 will be described below. In the present embodiment, program audio and dialog audio are handled as digital signals, and these signals are sampled (sampled) at an appropriate sampling frequency on the time axis. The following processing is premised on frame processing in which a predetermined number of frequency samples are collected in a predetermined unit time. In this embodiment, the program sound is described as a mono signal of 1ch. However, the present invention is not limited to this, and the present invention can also be applied when the program sound is 2ch or more.

  The determination unit 7 calculates the frame energy Ed of the dialog sound acquired from the second reception unit 5. The frame energy Ed [dB] of the dialog voice is given by the following equation, for example.

ここで、d(n)はダイアログ音声の時間表現であり、abs()は絶対値、maxは最大値をとる関数である。またｍは、log(0)を防ぐための、ｄ(ｎ)に比べて非常に小さな値である。フレームエネルギーＥｄは、フレーム内のダイアログ音声のエネルギーの最大値を表し、判断部７は、Ｅｄの値が所定の値よりも大きいフレームを、有意なエネルギーを有するフレームと判断する。本実施形態では、Ｅｄ＞−２０[dB]を満たしたフレームを、有意なエネルギーを有すると判断する。これにより、ダイアログ音声に含まれ得る、無音の時間（“ま”）からなるフレーム又は十分なエネルギーを有しないフレームを用いてゲイン量が推定されることを防ぐことができる。

Here, d (n) is a time expression of dialog voice, abs () is an absolute value, and max is a function that takes a maximum value. Further, m is a very small value compared to d (n) for preventing log (0). The frame energy Ed represents the maximum value of the dialog voice energy in the frame, and the determination unit 7 determines that a frame having a value of Ed larger than a predetermined value is a frame having significant energy. In the present embodiment, it is determined that a frame that satisfies Ed> −20 [dB] has significant energy. As a result, it is possible to prevent the gain amount from being estimated using a frame consisting of a silent time (“ma”) or a frame that does not have sufficient energy, which can be included in the dialog voice.

  Further, the determination unit 7 obtains the similarity between the program sound and the dialog sound of the same frame. In this embodiment, the determination part 7 calculates | requires the amplitude frequency characteristic of each program audio | voice and dialog audio | voice using a discrete Fourier transform (FFT) as a filter bank. The amplitude frequency characteristic D (k) of the dialog sound and the amplitude frequency characteristic B (k) of the program sound are given by the following equations.

ここで、ｄ(n)はダイアログ音声の時間表現、ｂ(n)は番組音声の時間表現であり、ｋは周波数上のサンプリングインデックスを表す。またFFT()は離散フーリエ変換を表し、abs()は絶対値をとる関数である。

Here, d (n) is a time expression of dialog sound, b (n) is a time expression of program sound, and k represents a sampling index on the frequency. FFT () represents a discrete Fourier transform, and abs () is a function that takes an absolute value.

  Further, in the present embodiment, the determination unit 7 divides the amplitude frequency characteristic B (k) of the program audio and the amplitude frequency characteristic D (k) of the dialog audio into a plurality of bands, respectively, Calculate the number of relationships. Note that since the dialog voice has energy concentrated in a lower band than a general voice signal, in this embodiment, only a part of the band in the transmitted program voice is calculated. Here, assuming that the number of samples in each frame is 2048, what is expressed by frequency by discrete Fourier transform can be expressed by 1024 samples. In the present embodiment, the time sampling frequency is set to 48 kHz, and the upper limit of calculation is set to 5 kHz for simplification, thereby calculating about 200 frequency samples among the 1024 frequency samples of each frame. In this embodiment, an example of dividing a band of 200 frequency samples or less into four is shown, but the present invention is not limited to this, and the band may be divided by more or less than four divisions. It is also possible to calculate without dividing. The amplitude frequency characteristic D (k) of the dialog sound and the amplitude frequency characteristic B (k) of the program sound are divided into four for each band by the following formula.

さらに判断部７は、Ｄ(k)及びＢ(k)の帯域成分ごとの相関関数の最大値ＭＣを以下の式により求める。

Further, the determination unit 7 obtains the maximum value MC of the correlation function for each band component of D (k) and B (k) by the following formula.

上式（５）のＭＣが１に近いほど、番組音声の帯域成分はダイアログ音声の同じ帯域成分と類似していることを示す。本実施形態においては、ＭＣ＞0.9を満たしたフレームを類似度が高いと判断する。

As MC in the above formula (5) is closer to 1, it indicates that the band component of the program sound is similar to the same band component of the dialog sound. In the present embodiment, it is determined that a frame satisfying MC> 0.9 has a high similarity.

  The determination unit 7 determines the validity of each frame based on the frame energy Ed of the dialog sound and the similarity between the program sound and the dialog sound. That is, the determination unit 7 determines that a frame in which the above-described frame energy Ed [dB] satisfies Ed> −20 and the maximum value MC of the correlation function for each band component satisfies MC> 0.9 is valid. Then, the determination unit 7 sends the program sound and dialog sound of the frame determined to be valid to the estimation unit 9.

  In the present embodiment, the processing in the determination unit 7 is performed using the time expressions b (n) and d (n) of the program audio and the dialog audio, but the first reception unit 3 and the second reception are performed. When the program audio and dialog audio before being decoded by the unit 5 are expressed in the frequency domain, the program audio and dialog audio expressed in the frequency domain are used as they are to perform the processing in the determination unit 7 Is also possible.

  The estimation unit 9 estimates the gain amount G of the dialog sound included in the program sound based on the program sound and the dialog sound of the frame determined to be valid by the determination unit 7. Here, the gain amount G is a value at which the same level as the dialog sound included in the program sound of the frame can be obtained by multiplying the dialog sound of a certain frame by the gain amount G. The gain amount G of the dialog sound included in the program sound is given by the following equation.

  In addition, the estimation unit 9 sets the gain amount G of the frame that is not determined to be valid by the determination unit 7 to be the same as the gain amount G of the previous frame. In this embodiment, the estimation unit 9 sets the gain amount G of the frame to 1 when the determination unit 7 does not determine that the first frame is valid. The estimation unit 9 sends the gain amount G of each frame to the adjustment unit 11.

  The adjustment unit 11 adjusts the dialog sound acquired from the second reception unit 5 based on the gain amount G acquired from the estimation unit 9. That is, in the present embodiment, the adjusting unit 11 adjusts the level of the dialog sound by dephasing the dialog sound acquired from the second receiving unit 5 and multiplying by the gain amount G. Therefore, in the present embodiment, the dialog sound adjusted by the adjusting unit 11 is an antiphase signal of the dialog sound included in the program sound. The adjustment unit 11 sends the adjusted dialog voice to the synthesis unit 13.

  The synthesizing unit 13 synthesizes the adjusted dialog sound acquired from the adjusting unit 11 and the program sound acquired from the first receiving unit 3. In the present embodiment, since the dialog sound adjusted by the adjustment unit 11 is an antiphase signal of the dialog sound included in the program sound, the dialog sound adjusted by the synthesis unit 13 and the program sound are included in the program sound. Dialogue audio will be canceled out. Therefore, the synthesizing unit 13 can output the dialog sound-removed program sound (that is, sound other than the dialog sound) from which the dialog sound is removed from the program sound, for example, to a sound output unit (not shown). When the program sound is 2ch or more, the dialog sound can be removed by applying the above method to each channel.

  Next, processing of the receiving device 1 will be described using FIG. FIG. 2 is a flowchart showing processing of the receiving device 1 according to the first embodiment of the present invention.

  First, the 1st receiving part 3 receives the broadcast signal broadcast by the broadcasting station, takes out a program audio | voice from a broadcast signal, and sends it to the judgment part 7 and the synthetic | combination part 13 for every flame | frame (step S101). Then, the second receiving unit 5 receives the dialog sound (communication signal) included in the program sound from the broadcasting station through the Internet communication, and sends it to the determination unit 7 and the adjustment unit 11 for each frame (step S102).

  Then, the determination unit 7 determines the validity of each frame of the program sound and the dialog sound based on the frame energy of the dialog sound and the similarity between the program sound and the dialog sound, and the program of the frame determined to be valid The voice and dialog voice are sent to the estimation unit 9 (step S103).

  Then, the estimation unit 9 estimates the gain amount G of the dialog sound included in the program sound based on the program sound and dialog sound of the frame determined to be valid, and the gain amount G of the frame not determined to be effective is The gain amount G of each frame is sent to the adjustment unit 11 as the same as the gain amount G of the previous frame (step S104).

  Then, the adjustment unit 11 dephases the dialog sound acquired from the second reception unit 5 and multiplies the dialog sound by the gain amount G, adjusts the dialog sound, and sends the dialog sound to the synthesis unit 13 (step S105).

  Then, the synthesizing unit 13 synthesizes the program audio acquired from the first receiving unit 3 and the adjusted dialog audio acquired from the adjusting unit 11 (step S106). The synthesizer 13 outputs the dialog sound removal program sound.

  As described above, in the present embodiment, the determination unit 7 determines the validity of each frame based on the program sound and the dialog sound, and the estimation unit 9 is based on the program sound and the dialog sound of the frame determined to be valid. Thus, the gain amount G of the dialog sound included in the program sound is estimated. That is, since the receiving apparatus 1 according to the present embodiment estimates the gain amount G only for valid frames, the gain amount G with high reliability can be obtained. As a result, it is possible to adjust the level of only the dialog sound included in the received broadcast signal.

  In the receiving apparatus 1 according to the present embodiment, the determination unit 7 determines the validity of each frame based on the frame energy of the dialog sound and the similarity between the program sound and the dialog sound. That is, the gain amount G is not estimated with respect to a frame of silent or minute level audio among the dialog audio to be level-adjusted, the dialog audio has sufficient energy, and the program audio and the dialog audio Since the gain amount G is estimated only for frames that are sufficiently similar to each other, a more reliable gain amount G can be obtained, and the level of the dialog sound included in the program sound of the broadcast signal is more strict. It becomes possible to adjust to. Also, the receiving apparatus 1 in the present embodiment divides the program audio and dialog audio bands into a plurality of bands, and determines that a frame having similarity in at least one band is valid, thereby providing a more reliable gain amount. G can be obtained, and the level of the dialog sound included in the program sound of the broadcast signal can be adjusted more strictly. In addition, the receiving apparatus 1 according to the present embodiment is more reliable by the estimation unit 109 making the gain amount G of a frame that is not determined to be valid the same as the gain amount G of the previous frame. It is possible to adjust the level of the dialog sound using the gain amount G. Furthermore, in the receiving apparatus 1 according to the present embodiment, the adjusting unit 11 adjusts the dialog sound by dephasing the dialog sound and multiplying by the gain amount G, and synthesizes the dialog sound adjusted by the combining unit 13 and the program sound. By doing so, it is possible to appropriately delete only the dialog sound included in the program sound whose level changes with time.

(Second Embodiment)
In the first embodiment, the case where the dialog sound removal program sound is output as one of the output signals of the receiving apparatus 1 has been described. However, in the second embodiment, a case where a further signal is added to the dialog sound removal program sound. explain.

  FIG. 3 is a functional block diagram showing a schematic configuration of the receiving apparatus 101 according to the second embodiment of the present invention. The receiving apparatus 101 of this embodiment includes a first receiving unit 103, a second receiving unit 105, a determining unit 107, an estimating unit 109, a first adjusting unit 111, a first combining unit 113, and a second adjustment. Unit 112 and a second combining unit 114. The functions of the first receiving unit 103, the determining unit 107, the estimating unit 109, the first adjusting unit 111, and the first combining unit 113 correspond to the first receiving unit 3, the determining unit 7, and the estimating unit of the corresponding first embodiment. 9, since it is the same as the function of the adjustment part 11 and the synthetic | combination part 13, description is abbreviate | omitted.

  The second receiving unit 105 in the present embodiment receives a communication signal from a broadcasting station via the Internet, and sends the received dialog sound to the determination unit 107, the first adjustment unit 111, and the second adjustment unit 112.

  The second adjustment unit 112 multiplies the dialog voice acquired from the second reception unit 105 by a predetermined value for adjustment, and sends the adjusted dialog voice to the second synthesis unit 114.

  The second synthesis unit 114 synthesizes the adjusted dialog sound acquired from the second adjustment unit 112 and the dialog sound removal program sound acquired from the first synthesis unit 113. As a result, it is possible to output the program sound obtained by replacing the dialog sound subjected to the level change process included in the program sound with the dialog sound appropriately adjusted by the second adjustment unit 112. Note that the predetermined value multiplied by the dialog voice in the second adjustment unit 112 may be a value set in advance by the receiving device 101, or the receiving device 101 includes an input unit such as a volume knob, for example. It can also be a value appropriately input by the user by operating the input unit.

  Subsequently, processing of the reception apparatus 101 will be described with reference to FIG. FIG. 4 is a flowchart showing processing of the receiving apparatus 101 according to the second embodiment of the present invention.

  First, similarly to step S101 in the first embodiment, the first receiving unit 103 receives a broadcast signal broadcast by a broadcasting station, extracts program audio from the broadcast signal, and determines the determination unit 107 and the first combining unit for each frame. 113 (step S201). Then, the second receiving unit 105 receives the dialog sound (communication signal) included in the program sound from the broadcasting station by Internet communication, and determines the determination unit 107, the first adjustment unit 111, and the second adjustment unit 102 for each frame. (Step S202).

  Then, as in step S103 in the first embodiment, the determination unit 107 determines the validity of each frame of the program audio and the dialog audio based on the frame energy Ed of the dialog audio and the similarity between the program audio and the dialog audio. The frame determined to be valid for the program audio and the dialog audio is sent to the estimation unit 109 (step S203).

  Then, as in step S104 in the first embodiment, the estimation unit 109 estimates the gain amount G of the dialog sound included in the program sound based on the program sound and the dialog sound of the frame determined to be valid, and The gain amount G of the frame that has not been determined is assumed to be the same as the gain amount G of the previous frame, and the gain amount G of each frame is sent to the first adjustment unit 111 (step S204).

  Then, the first adjustment unit 111 adjusts the dialog voice by dephasing the dialog voice and multiplying it by the gain amount G, and sends it to the first synthesis unit 113 (step S105), as in step S105 in the first embodiment. S205).

  Then, the first synthesizing unit 113 synthesizes the program audio acquired from the first receiving unit 103 and the adjusted dialog audio acquired from the first adjusting unit 111 (step S106 in the first embodiment). S206).

  Then, the second adjustment unit 112 adjusts the dialog sound by multiplying the dialog sound by a predetermined value, and sends the dialog sound to the second synthesis unit 114 (step S207).

  Then, the second synthesizing unit 114 synthesizes the dialog sound removing program sound synthesized by the first synthesizing unit 113 and the adjusted dialog sound acquired from the second adjusting unit 112 (step S208). The second synthesizing unit 114 outputs the program sound obtained by replacing the dialog sound subjected to the level change process included in the program sound with the dialog sound appropriately adjusted by the second adjusting unit 112.

  As described above, the receiving apparatus 101 according to the present embodiment deletes the dialog sound included in the received broadcast signal, and further synthesizes the dialog sound appropriately adjusted by the second adjustment unit 112, so that the dialog at a desired level is obtained. It is possible to output program audio including audio.

  Thus, according to the present invention, it is possible to provide a receiving apparatus and a program that can adjust the level of only dialog sound included in the received broadcast signal.

DESCRIPTION OF SYMBOLS 1,101 Receiving device 3,103 1st receiving part 5,105 2nd receiving part 7,107 Judgment part 9,109 Estimating part 11,111 Adjustment part (1st adjustment part)
13, 113 synthesis unit (first synthesis unit)
112 2nd adjustment part 114 2nd synthetic | combination part

Claims (7)

A first receiver for receiving broadcast program audio;
A second receiver for receiving the dialog voice of the program voice by Internet communication;
A determination unit that determines the validity of each frame based on the program sound and the dialog sound;
An estimation unit configured to estimate a gain amount of a dialog sound included in the program sound based on the program sound and the dialog sound of the frame determined to be valid by the determination unit;
An adjusting unit for adjusting the dialog sound based on the gain amount;
A receiving device comprising: a synthesizing unit that synthesizes the dialog sound adjusted by the adjusting unit and the program sound.   The receiving apparatus according to claim 1, wherein the determination unit determines the validity of each frame based on frame energy of the dialog sound and similarity between the program sound and the dialog sound.   The receiving device according to claim 2, wherein the determination unit divides a band of the program audio and the dialog audio into a plurality of frames and determines that a frame having similarity in at least one band is valid.   The receiving apparatus according to any one of claims 1 to 3, wherein the estimation unit makes the gain amount of a frame that is not determined to be valid the same as the gain amount of the previous frame.   The receiving device according to any one of claims 1 to 4, wherein the adjustment unit adjusts the dialog voice by dephasing the dialog voice and multiplying the gain by the gain amount. 6. The apparatus according to claim 1, further comprising: a second synthesis unit that further synthesizes the result synthesized by the synthesis unit and the dialog voice different from the dialog voice among the dialog voices received by the second reception unit. The receiving device according to item. In the computer of the receiving device,
Obtaining broadcast program audio; and
Obtaining a dialog sound of the program sound by Internet communication;
Determining the validity of each frame based on the program audio and the dialog audio;
Estimating a gain amount of the dialog sound included in the program sound based on the program sound and the dialog sound of the frame determined to be valid;
Adjusting the dialog sound based on the gain amount;
A program for executing the step of synthesizing the dialog sound and the program sound adjusted in the adjusting step .

Images