JP2005045739A - Apparatus, method and system for telephone conversation - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To dynamically determine an acoustic frequency component to be encoded/transmitted in accordance with a network band without cutting off a telephone conversation when performing high-quality voice communications combining BGM (back ground music) and sound effect or the like with voice via the Internet. <P>SOLUTION: A VoIP (voice over Internet protocol) client applies high-efficiency compression encoding to data resulting from combining BGM or effective sounds with voice, makes the result into RTP (real-time transport protocol) packet and exchanges it with the other VoIP client who becomes a communicating party, to perform the telephone conversation. This VoIP client transmits to the other VoIP client an RTCP (real-time control protocol) APP packet describing control information such as an encode bandwidth designating the band to be encoded by the other VoIP client and the number of sub-band divided blocks resulting from dividing the encode bandwidth. The other VoIP client who receives this RTCP packet, changes a CODEC (coder-decoder) on the basis of the control information and transmits the RTP packet. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a call device, a call method, and a call system that perform a so-called Internet telephone using, for example, VoIP (Voice over Internet Protocol), and is particularly suitable for exchanging real-time acoustic data composed of high-quality voice and sound. TECHNICAL FIELD The present invention relates to a simple call device, a call method, and a call system.

  VoIP is a technology that enables voice communication via an IP network by encapsulating voice in IP packets. In order to make a call by VoIP, it is necessary to exchange a series of information such as acquisition of information on the other party to be called, calling of the other party to be called, and response. For these purposes, SIP (Session Initiation Protocol) etc. A call control protocol is used.

  In recent years, systems that transmit real-time data such as real-time moving image data and real-time acoustic data such as audio data by using a communication network such as the Internet including such VoIP are increasing. In such a public line network such as the Internet, since a plurality of users supply the network bandwidth, a congestion control method, that is, a congestion avoidance method and a sedation method at the time of occurrence of congestion is a major issue. . Therefore, with a change in the number of communication modes in which real-time property is important, congestion control techniques in real-time data communication have become important (see Patent Document 1 below).

  By the way, conventionally, in real-time voice communication using VoIP, the main information of human voice is included in the frequency band near 4 to 5 KHz, so the number of channels is 1 and the sampling frequency is 8 KHz. It was usual to set to ~ 16KHz. However, it is difficult to transmit background noise and atmosphere in such a narrow band.

  Therefore, if the number of channels is set to 2 or more and the sampling frequency is raised to 44.1 KHz, which is the same level as CDDA (Compact Disk Digital Audio), background noise and atmosphere can be transmitted. It is also possible to add a high-quality BGM function using music content.

  However, when the number of channels is set to 2 or more, the sampling frequency is raised to 44.1 KHz, which is the same level as CDDA, and uncompressed 16-bit Linear PCM (Phase Code Modulation) data is transmitted, the transmission bit on the network The rate is over 1411.2Kbps, making it difficult to use in a real internet environment. Therefore, it is necessary to use a high-efficiency encoding method at present, but with the recent improvement in CPU performance, it has become possible to adopt a high-efficiency encoding method even in real-time voice communication.

  There are several high-efficiency encoding methods. First, frequency decomposition is performed, the signal is divided into a plurality of subbands, and each block is encoded using the psychoacoustic characteristics. Most of them adaptively change and entropy code a predetermined frequency component with a minimum number of bits. With this high-efficiency encoding, the bit rate becomes variable according to the frequency component to be encoded.

  That is, when the transmission bit rate and the sound quality are low as in the case of the conventional general telephone, it is not necessary to dynamically control the congestion by changing the compression method parameters according to the network conditions. In the case of performing real-time voice communication for exchanging high-quality real-time acoustic data with the addition of the BGM function or the like, the data transmission rate of the network becomes high. Therefore, when the network bandwidth starts to be congested for some reason during a call, the call is often forced to be temporarily interrupted, and high-quality sound with such a BGM function added. In the real-time communication, it is necessary to dynamically change the parameters of the compression method as in Patent Document 1 described above.

Japanese Patent Application Laid-Open No. 2001-3200430

  However, in real-time voice communication by VoIP, basically, signaling is performed using a call control protocol such as SIP to know the IP address, port number, voice and acoustic compression method of the other party. The basic mechanism is to send and receive RTP (Real-Time Transport Protocol) and RTCP (Real-Time Control Protocol) streams bi-directionally between itself and its target address. Normally, once an RTP stream is established, the audio compression method cannot be changed unless the recall control is performed. Further, when the recall control is performed, the call is temporarily interrupted.

  In other words, if it is classified as a separate voice / music compression method according to the frequency component to be encoded, it is necessary to perform recall control. Therefore, when the network bandwidth is congested for some reason, the transmission bit rate remains unchanged. Even so, there is a problem that the call is temporarily interrupted even by the recall control required to lower the transmission bit rate. Therefore, it is desired to generalize a mechanism in which the same voice / music compression method dynamically determines the number of subband divisions and frequency components to be encoded without blocking a call according to the network band.

  The present invention has been proposed in view of such a conventional situation, and when performing high-quality voice communication in which voice is synthesized with BGM, sound effects and the like via the Internet, the call is not interrupted. Another object of the present invention is to provide a call device, a call method, and a call system capable of dynamically determining an acoustic frequency component to be encoded / transmitted according to a network band.

  In order to achieve the above-described object, a communication device according to the present invention includes a compression encoding unit that compresses and encodes real-time acoustic data including voice in a communication device that transmits and receives real-time acoustic data via the Internet. A data packet generating means for generating a data packet storing data compressed and encoded by the compression encoding means, and a control packet generating means for generating a control packet storing at least management information for managing transmission / reception of the data packet Transmitting means for transmitting the data packet and the control packet to one or more other communication devices; receiving means for receiving the data packet and control packet from the other communication device; and the other communication Control means for generating control information for controlling the transmission rate of data packets from the device, and the control Packet generating means, and generates the control packet that stores the control information together with the management information.

  In the present invention, the control information for controlling the transmission rate of the data packet from the other communication device is generated, and this is received by transmitting it to the other communication device as the control packet stored together with the management information. It is possible to set the transmission rate of the data packet transmitted by the other call device based on the control information of the control packet. In other words, when the network bandwidth starts to be suddenly congested or when it is difficult to hear on the receiving side, the data packet receiving side can control the transmission rate of transmission data on the data packet transmitting side. In addition, since this control information is transmitted together with the control packet instead of the message at the time of call control, it is not necessary to stop the transmission / reception of the data packet.

  Further, the control means measures the transmission rate of the data packet from the other call device received by the receiving means, and generates the control information based on the measurement result, or the control information based on a user operation. As the control information, information for controlling the compression rate of the real-time acoustic data stored in the data packet from the other call device, for example, the data packet from the other call device is stored. Information specifying the frequency band to be compressed and / or the number of subband divisions of the real-time acoustic data to be compressed. Thereby, the compression encoding system of the real-time acoustic data stored in the data packet transmitted by another telephone apparatus can be changed.

  

  Furthermore, the real-time acoustic data stored in the data packet is a combination of voice and background music and / or sound effects, and not only the voice but also background music is added to reduce the amount of information. Even if the transmission data is large, communication is not interrupted by controlling the transmission rate as necessary.

  The compression encoding means changes a compression rate of real-time acoustic data to be transmitted to the other call device based on control information for controlling the transmission rate stored in a control packet from the other call device. It is possible to change the compression rate of real-time acoustic data stored in a data packet to be transmitted in response to a request from another telephone device that is a call partner, and to change the transmission rate of the data packet to be transmitted .

  The call method according to the present invention is a call method for transmitting and receiving real-time acoustic data via the Internet. A compression encoding step for compressing and encoding real-time acoustic data including speech, and compression by the compression encoding step. A data packet generating step for generating a data packet storing encoded data, a control packet generating step for generating a control packet storing at least management information for managing transmission / reception of the data packet, and one or more other calls A transmission step of transmitting the data packet and the control packet to a device; a reception step of receiving the data packet and the control packet from the other communication device; and a transmission rate of the data packet from the other communication device. And a control step for generating control information for controlling the control packet. And generating the control packet that stores the control information together.

  The call system according to the present invention is a call system in which two or more call devices transmit and receive real-time acoustic data via the Internet. Each call device compresses and encodes real-time acoustic data including voice. Means, a data packet generating means for generating a data packet storing data encoded by the compression encoding means, and a control packet for generating a control packet storing at least management information for managing transmission / reception of the data packet Generating means, transmitting means for transmitting the data packet and the control packet to one or more other communication devices, receiving means for receiving the data packet and control packet from the other communication device, and the other Control means for generating control information for controlling the transmission rate of data packets from the telephone device The control packet generation means of one of the at least two call devices generates the control packet storing the control information for controlling the transmission rate of the data packet from the other call device together with the management information, and the other call The compression encoding means of the apparatus changes the compression rate of the real-time acoustic data transmitted to the one communication device based on the control information stored in the control packet from the one communication device. .

  In the present invention, one call device can control the transmission rate of data packets transmitted by the other call device that is the subject of the call, and the other call device can respond to a request from the other call device. It is possible to control the transmission rate of data packets transmitted to the other telephone device by changing the compression encoding method of the real-time acoustic data transmitted to the other telephone device.

  According to the communication device according to the present invention, in the communication device that transmits and receives real-time acoustic data via the Internet, the compression encoding means that compresses and encodes real-time acoustic data including voice, and the compression encoding means A data packet generating means for generating a data packet storing the compression-encoded data, a control packet generating means for generating a control packet storing at least management information for managing transmission / reception of the data packet, and one or more other data packets Transmission means for transmitting the data packet and the control packet to the communication device, reception means for receiving the data packet and control packet from the other communication device, and transmission of the data packet from the other communication device Control means for generating control information for controlling the rate, the control packet generation means, Since the control packet storing the control information together with the physical information is generated, the data packet receiving side controls the transmission transmission rate of the other communication device on the data packet transmitting side by the control packet. For example, if the network bandwidth suddenly starts to get congested or if it is difficult to hear on the receiving side, the transmission rate transmitted by the other call device that is the other party of the call is reduced without re-controlling the call. Therefore, even when real-time acoustic data having a high sound quality and a large amount of information is transmitted / received, the call can be continued.

  Further, according to the call system of the present invention, in a call system in which two or more call devices transmit and receive real-time acoustic data via the Internet, each call device compresses and encodes real-time sound data including voice. Compression encoding means for generating data packets, data packet generating means for generating data packets storing data encoded by the compression encoding means, and control packets storing at least management information for managing transmission / reception of the data packets. Control packet generating means for generating, transmitting means for transmitting the data packet and the control packet to one or more other telephone apparatuses, and receiving means for receiving the data packet and control packet from the other telephone apparatus And control for generating control information for controlling the transmission rate of data packets from the other communication device. The control packet generation means of one of the at least two communication devices generates the control packet storing the control information for controlling the transmission rate of the data packet from the other communication device together with the management information. Then, the compression coding means of the other call device changes the compression rate of the real-time acoustic data transmitted to the one call device based on the control information stored in the control packet from the one call device. Therefore, one call device can control the transmission rate of data packets from the other call device that is the subject of the call, and can respond to the request from the other call device to the other call device. The transmission rate can be controlled by changing the compression encoding method of the real-time acoustic data stored in the data packet to be transmitted, and the amount of information is large, such as BGM or Since the transmission rate can be changed dynamically even when real-time sound data with high quality sound with added sound etc. is exchanged, the call is blocked even when the network bandwidth is congested There is nothing.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In this embodiment, the present invention is applied to a VoIP client as a call device in which two or more call devices make a call by VoIP over the Internet and a VoIP system as a call system including the same. The VoIP client according to the present embodiment makes the voice quality variable according to the network bandwidth, that is, the number of subband divisions and the encoding in the same voice / music compression system, in order to enable the call to continue between the call devices. The power frequency component is dynamically determined.

  First, an outline of a VoIP system that performs network communication using VoIP in the present embodiment will be described. FIG. 1 is a schematic diagram showing an example of a VoIP system in the present embodiment. The VoIP system according to the present embodiment realizes high-quality voice communication that can share not only a call but also various sound effects and BGM, for example, between two or more channels and between users. Note that the VoIP system in the present embodiment is performed between two call devices (hereinafter referred to as VoIP clients). However, the number of VoIP clients constituting the VoIP system is not limited to two. There may be two or more participants who can participate in network communication.

  As shown in FIG. 1, the VoIP system 100 includes a VoIP client 111 such as a PC (Personal Computer), for example, and a VoIP client 121 connected thereto via the Internet 130.

  In the VoIP system 100, the user 110 of the VoIP client 111 and the user 120 of the VoIP client 121 are VoIP applications (soft phones), which will be described later, installed in their VoIP clients 111 and 121, for example. A headset composed of a microphone and headphones or a handset composed of a microphone and a receiver is used to communicate with a communication partner via the Internet 130.

  The Internet 130 is a network environment that is spread all over the world by connecting a plurality of communication lines such as general public lines and information communication networks. Currently, broadband transmission is enabled by the widespread use of broadband and high-speed communication lines. A network is configured with a communication line of 500 kbps or higher using an optical fiber, an asymmetric digital subscriber line, radio, or the like.

  Connected to the Internet 130 are a VoIP server 131 for controlling VoIP communication, a web server 134 for managing data such as sound source data 132 and download user information 133, and the like. Each VoIP client 111, 121 stores sound source data 113, 123 downloaded from the web server 134 by the web browser 112, 122 or the like included in each VoIP client 111, 121, or purchased or edited by itself. Yes.

  The sound source data 132 stored in the database of the web server 134 and the sound source data 113 and 123 possessed by the user by downloading, for example, are music used for BGM (Back Ground Music) or the like, or sound of waves -Various sound effects such as clapping sound, thunder, bell sound, etc. These sound source data can be used in network communication between users.

  Next, the VoIP client will be described. FIG. 2 is a block diagram showing the functions of the VoIP client constituting the VoIP system. As shown in FIG. 2, the VoIP client 20 for performing such Internet communication receives the data from the calling party and the transmission means 21 for transmitting data to the participants participating in the communication, that is, the calling party. Receiving means 41.

  The transmission means 21 is connected to a microphone, and is compressed into a microphone capture (MIC capture) 22 that catches the external (user) voice and, for example, MP3 (MPEG (Moving Picture Experts Group) 1 audio Layer 3) or MPEG4 Sound effect file storage section 23 storing sound source files of various sound effects (SE), BGM file storage section 24 storing sound source files of various compressed BGM, sound effect files and BGM files Decoders 25 and 26 that respectively read and decode the sound, gain adjusters 27 to 29 that control the gain of the sound, sound effects, and BGM captured by the microphone, and a synthesizer 30 that synthesizes these three sounds. An encoder 31 that compresses and encodes the synthesized sound; A packetizing unit 32a as a data packet generating means to be stored in an RTP packet as a packet, a control unit 34 for generating control information to be described later, management information for controlling the control information and the RTP packet, etc. And an RTCP packetizing unit 32b as a control packet generating means to be stored in and a transmitting unit 33 for transmitting RTP packets, RTCP packets, and the like. The RTP packet sent from the sending unit 33 is sent via the Internet 130 to another VoIP client to be communicated.

  On the other hand, the receiving means 41 for receiving data sent from the transmitting means 21 of the communication partner via the Internet 130 includes a receiving section 42 for receiving RTP packets and RTCP packets, and an RTP depacketizing section for depacketizing the received RTP packets. (Depacketize) 43a, an RTCP depacketizing unit (depacketize) 43b for depacketizing an RTCP packet, a dejitter unit (de-jitter) 44 for correcting the arrival time of the RTP packet, and a portion where an error has occurred in the sent RTP packet Alternatively, a packet compensator 45 that compensates for missing parts such as a lost part during transmission, a decoder 46 that decodes and decompresses data from the packet compensator 45, and an incoming call that is compressed to MP3, MPEG4, etc. Ring tone file (Ring Tone) that stores sound source files File) storage unit 47, decoder 48 that reads and decodes the ringtone file, gain adjustment units 49 and 50 that adjust the gains of decoders 46 and 48, and synthesis unit 30 in transmission means 21 described above. PCM data for transmission, that is, a gain adjusting unit 52 that adjusts the gain of the transmitter's own sound source, and a combining unit that combines the gain-adjusted transmission data, data transmitted from the communication partner, and ringtone data 53, an output unit 54 that outputs the data synthesized by the synthesis unit 53 to the headphones (HP), a gain adjustment unit 51 that adjusts the gain of the ringtone separately from the ringtone output to the synthesis unit 53, And a speaker (SP) 55 for outputting a gain-adjusted ring tone to the outside.

  Next, a data transmission / reception method of the VoIP client will be described. First, on the transmission side, the microphone capture 22 catches the user's voice input by the microphone and sends it to the gain adjustment unit 27. The gain adjusting unit 27 multiplies the audio data by a gain coefficient k1 in accordance with a user instruction or automatically, and adjusts the gain to a desired magnitude.

  The sound effect file storage unit 23 and the BGM file storage unit 24 store, for example, a hard disk drive (HDD), a ROM (read only memory), or a magneto-optical device in which a sound source file compressed in advance by a compression technique such as MP3 or MPEG4 is stored. It consists of a disk, and the decoders 25 and 26 read out these compressed encoded data, and convert the read compressed encoded data into PCM data. Furthermore, the decoders 25 and 26 send the converted PCM data to the gain adjustment units 28 and 29, respectively, and the gain adjustment units 28 and 29 respectively send the received data according to a user instruction or automatically. Multiply the gain coefficients k2 and k3 to adjust the gain to a desired size.

  The sound, BGM, and sound effect of the real-time acoustic data whose gain is adjusted by the gain adjusting units 27 to 29 are supplied to the synthesizing unit 30, and the synthesizing unit 30 adds the outputs of the gain adjusting units 27 to 29 while performing saturation processing. The addition result is output to the encoder 31. The encoder 31 encodes, for example, MPEG4 in order to transmit the addition result through the network. The encoded data is supplied to an RTP packetizing unit 32 that packetizes the data according to a real-time transport protocol (RTP).

  The RTP packetization unit 32a packetizes the encoded data into RTP packets, and the packetized data is transmitted from the transmission unit 33 to the VoIP client of the communication partner via the Internet 130.

  In addition, the control unit 34 manages various management information stored in a report block of an RTCP packet, which will be described later, and generates control information stored in an extended portion of the RTCP packet. Control information is information specifying the compression rate of real-time acoustic data stored in an RTP packet transmitted by another VoIP client as a communication partner, for example, information such as the encoding band and the number of subband divisions, Information indicating a retransmission request for a lost packet in which an error occurs or is lost. Also, in the extended portion of the RTCP packet transmitted and received by the VoIP client in the present embodiment, a fluctuation absorbing buffer for canceling the jitter of the packet received by the RTCP receiving unit 42 and uncompressed PCM data before reproduction are stored. The capacity of the PCM buffer to be stored can also be described.

  The RTCP packetizing unit 32b packetizes the control information and management information into RTCP packets, and the packetized data is also transmitted from the transmitting unit 33 to the VoIP client of the communication partner via the Internet 130.

  In a VoIP call, before transmitting an RTP packet, the control unit 34 performs call signaling to another VoIP client to be communicated via the transmission unit 33 using a call control protocol such as SIP. Then, after the RTP / RTCP session is established between other VoIP clients, the transmission unit 33 can transmit the RTP packet and the RTCP packet. Details of the call signaling method and the RTP / RTCP packet will be described later.

  On the other hand, on the receiving side, the receiving unit 42 receives RTP packets, RTCP packets, and the like sent via the Internet 130. Then, the RTCP depacketization unit 43b disassembles the received RTCP packet, and the control unit 34 extracts control information stored in the extended portion of the RTCP packet, and among the control information, the above-described encoding band and subband are extracted. Compression rate control information including the number of divisions is supplied to the encoder 31. The encoder 31 changes the compression rate of the real-time acoustic data to be encoded according to the compression control information as a request from another VoIP client that is a call partner.

  Further, the depacketizing unit 43 converts the RTP packet received by the receiving unit 42 into a reverse packet, and then receives data whose arrival times are unequal because the dejittering unit 44 delays reception due to, for example, the state of the network. Correct and equalize. This correction is performed based on the RTP time stamp and sequential number that are depacketized and decomposed from IP and UDP (User Datagram Protocol). Thereafter, the packet compensator 45 ensures the packet that is lost or cannot be received in the transmission / reception of the network, specifically, the same packet as the previous or subsequent packet is used instead of the lost packet. The packet loss is compensated by requesting retransmission of the lost packet or receiving the lost packet again. The compressed data such as MPEG4 obtained in this way is decoded by the decoder 46, and the gain adjustment unit 49 multiplies the gain coefficient k5 by the user or automatically adjusts the gain.

  The decoder 48 generates a ring tone sound source file including a ring tone for calling or a music used for calling when a call is signaled from another VoIP client, that is, when a call is received. Read from the ringtone file storage unit 47. The ring tone data from the file storage unit 47 is selected in advance according to the user's request, and is decoded by the decoder 48 while being read into a RAM (not shown) according to the timing of the incoming call. The sound file of the ringtone is also compressed to MP3, MPEG4, etc., like the sound file of BGM, etc., and is stored in the file storage unit 47 for a plurality of files as ringtone data for each file. Then, the decoder 48 decodes the read sound source file, and gain adjustment is performed by the gain adjusting units 50 and 51 multiplying the decoded sound source data by a user instruction or automatically by gain coefficients k6 and k7, respectively.

  The synthesizing unit 53 receives the real-time acoustic data received from the call partner, which is transmitted from the ringtone and the communication partner and gain-adjusted by the gain adjusting unit 49, and the transmission transmitted by the gain adjusting unit 52 itself. The side real-time acoustic data is added. The gain adjustment unit 52 multiplies the transmission side real-time acoustic data by a gain coefficient k4 that is a loopback volume level set by the user in order to share the transmission side real-time acoustic data with the other party.

  The data synthesized by the synthesis unit 53 is output via the headphones 54 and transmitted to the user. In addition, the ring tone data read from the ring tone file storage unit 47 is configured to be output from the speaker 55 separately from the headphones 54.

  In such a VoIP client, control information is stored in an RTCP packet and transmitted to another VoIP client as a communication partner, thereby controlling a transmission rate of a received RTP packet according to a network congestion state. Can do. That is, it is possible to specify the encoding band and the number of subband divisions to be compressed by other VoIP clients, and to continue the call even if the sound quality temporarily decreases.

  Furthermore, by widening the dynamic range as compared with a general telephone, it is possible to transmit and receive a synthesized sound in which a high-quality sound such as stereo and sound composed of BGM and sound effects are synthesized. Even if is added, conversation (voice) is not masked. In addition, since the sound read from the sound source file such as the sound effect and BGM and the sound spoken by the communication person, that is, the microphone sound can be adjusted separately, the sound effect and the volume of the BGM can be set to desired levels. The user can adjust the user's feelings, for example, to the communication partner.

  In addition, by outputting the ringtone separately from the headphone 54 and the speaker 55, for example, the user temporarily leaves the VoIP client 20 during VoIP communication using the headphone 54, or even after the call ends. Even when 54 is left set, the ringtone can be output from the speaker 55 to the outside.

  Next, software used for the VoIP client will be described. FIG. 3 is a diagram illustrating a software module including a general VoIP client application for a PC. The VoIP client performs the function shown in FIG. 2 by executing the software module 1 corresponding to the protocol of each layer based on the open system interconnection (OSI) architecture shown in FIG. Achieve.

  In FIG. 3, each layer will be described from the lower layer to the upper layer. First, the functions as the physical layer 2 include a universal serial bus (USB) camera driver 2a, a USB audio driver 2b, and various drivers 2c. This is a layer that matches the physical conditions of the transmission conditions of video data from the camera driver 2a and audio data from the audio driver 2b.

  Next, the function as the data link layer includes an operating system (OS) 3. This is for executing data transmission without error between adjacent nodes.

  The network layer function includes an Internet Protocol (IP) control unit 4. The network layer selects a communication path used for data transmission / reception and performs communication control such as flow control and quality control. IP, which is a connectionless packet transmission protocol that does not pursue reliability, leaves the reliability assurance function, flow control function, and error recovery function to the upper layers (transport layer and application layer).

  The function as the transport layer includes a transport control protocol (TCP) / user datagram protocol (UDP) control unit 5.

  In the transport layer, end-to-end transmission is performed using an IP address. Regardless of the type of network, flow control and sequence control are performed according to the required quality class. TCP has a reliability guarantee function, attaches a sequence number to each byte of transmitted data, and retransmits the data if no acknowledgment (Acknowledgment: ACK) is sent from the receiving side. UDP provides a function for transmitting datagrams between applications. When streaming an audio / video using an IP network, generally, a transport protocol such as TCP that retransmits when an error occurs cannot be used. TCP is a protocol for one-to-one communication, and information cannot be transmitted to a plurality of partners. Therefore, UDP is used for such applications. In other words, when performing highly reliable communication such as requiring retransmission control, TCP packets are generated as TCP communication, and communication with high real-time properties such as transmission of audio and video data and SIP described later is performed. When performing, UDP communication is generated as UDP communication.

  UDP is designed to allow application processes to transmit data to other application processes on a remote machine with minimal overhead. Therefore, the information entered in the UDP header is only the source port number, destination port number, data length, and checksum, and there is no field for entering the number indicating the order of packets in TCP. When the order of the packets is changed due to transmission through the network, processing for returning the order to the correct state cannot be performed. Also, there is no field for inputting time information such as a time stamp at the time of transmission in TCP or UDP.

  The functions as the session layer 6 include a session initiation protocol (SIP) control unit 11, an RTP / RTCP control unit 12, a codec (codec) 13, speech sound and BGM or sound effect synthesis processing software. There are a holding tone control unit 14, a BGM synthesis unit 15, and a ringing tone control unit 16 which are configured. The session layer 6 controls information transmission. Manage conversation modes between applications and control conversation units.

  The SIP control unit 11 performs call control according to the session layer 6 signaling protocol (standardized by RFC3261) for establishing, changing, and terminating a multimedia session on an IP network.

  Of the RTP / RTCP control unit 12, RTP is a protocol for transmitting transmission data in which BGM and sound effects are combined with audio data and compressed and encoded. Time information and order information are added to the transmission data. And has a function of transmitting and receiving voice data through the network. RTCP is a control protocol for controlling RTP, and has functions of performing RTP flow control, clock synchronization, data reproduction time recognition, information source recognition, and the like.

The codec 13 has a function of compressing and decoding the transmission sound (transmission data) described later with high efficiency.
The VoIP call control 7 is a function as the presentation layer. The presentation layer manages the expression format of information transmitted and received by the application, and performs data conversion and encryption. The VoIP call control unit 7 is a control unit that controls the entire VoIP call function.

  As a function of the uppermost application layer, there is a graphical user interface (GUI) 8 for visually operating a computer. The application layer is a layer for managing external specifications of communication functions used in the user program and exchanging information based on the external specifications. The GUI 8 provides an interface for user operation and handles user input information. .

  FIG. 4 is a schematic diagram illustrating an example of a GUI. As shown in FIG. 4, the GUI 8 includes an application control unit 61 that performs termination processing of the VoIP client application 1, an information display unit 62, a dial unit 63, a headset volume unit 64a, a speaker volume unit 64b, and a sound. It has an effect selection display unit 65, a sound effect control unit 66, a BGM selection display unit 67, a BGM control unit 68, and a sound quality control unit 69.

  The information display unit 62 displays a dial number when the user dials the number of the other party to be communicated, a partner state such as whether or not the communication partner is busy, and the like. The dial unit 63 includes a numeric keypad for dialing a VoIP counterpart. The headset volume 64a adjusts the volume output from the headset, and the speaker volume unit 64b adjusts the volume output from the speaker. The sound effect selection unit 65 displays a sound effect sound source data file that can be used by the user. For example, a gunshot sound, lightning sound, applause sound, cheering sound, etc. can be selected. Play and stop sound, and adjust volume. Thereby, a user can express feelings to the other party with various sound effects.

  Further, the BGM selection display unit 67 displays a BGM sound source data file such as a sound of a Tahiti wave that can be used by the user. Further, the BGM control unit 68 further plays and stops the BGM and the volume of the BGM. By performing the adjustment, the user's mood and the atmosphere of the place can be communicated to the communication partner by the volume selected and adjusted by the user himself, like the sound effect.

  Furthermore, the sound quality control unit 69 allows the user to select the sound quality when performing only voice communication and when performing high sound quality communication with BGM and sound effects, so that the encoder can compress the compression code. The transmission rate is variable.

  Next, the hardware configuration of the VoIP client that executes the software module 1 will be described. FIG. 5 is a block diagram showing a hardware configuration of the VoIP client. As shown in FIG. 5, in the VoIP client 20, a CPU (Central Processing Unit) 221 includes various programs constituting the software module stored in a ROM (Read Only Memory) 222 or a RAM (Random) from the storage unit 228. Access Memory) 223 executes various processes in accordance with various programs constituting the software module 1 described above. In addition, a timer (not shown) performs timekeeping operation and supplies time information to the CPU 221. The RAM 223 also appropriately stores data necessary for the CPU 221 to execute various processes.

  The CPU 221, ROM 222, and RAM 223 are connected to each other via a bus 224. An input / output interface 225 is also connected to the bus 224.

  The input / output interface 225 includes an input unit 226 including a keyboard and a mouse, a display including a CRT and an LCD, an output unit 227 including a headphone and a speaker, a storage unit 228 including a hard disk, a modem, a terminal A communication unit 229 composed of an adapter or the like is connected.

  The communication unit 229 performs communication processing via the Internet (not shown). Data provided from the CPU 221 is transmitted. The communication unit 229 outputs data received from the communication partner to the CPU 221, RAM 223, and storage unit 228. The storage unit 228 communicates with the CPU 221 to store and delete information. The communication unit 229 also performs analog signal or digital signal communication processing with other clients.

  A drive 230 is connected to the input / output interface 225 as necessary, and a magnetic disk 241, an optical disk 242, a magneto-optical disk 243, a semiconductor memory 244, or the like is appropriately mounted, and a computer program read from these is loaded. It is installed in the storage unit 228 as necessary.

  Next, a method for executing a VoIP call by the VoIP client having such a hardware configuration executing various programs constituting the software module 1 shown in FIG. 2 will be described.

  FIG. 6 is a diagram for explaining the RTP / RTCP packet transmission / reception function controlled by the VoIP call control unit 7 in the general PC software module shown in FIG. As shown in FIG. 6, the control of the VoIP call control unit 7 can be divided into threads 1 to 5. The thread 1 is a process until the sender transmits an RTP packet, that is, a process of transmitting a data packet containing data including voice uttered by the sender, and the threads 2 and 3 transmit the transmitted RTP packet. Processing until reception and reproduction, that is, processing until the sender hears data including the voice of the other party with whom communication is made.

  The thread 4 is automatically generated by a thread library attached to the operating system (OS) 3. The thread library performs calculation resource allocation, that is, scheduling, on the main processor (CPU 221 shown in FIG. 5) of the thread 1, thread 2, and thread 3 according to priority.

  The thread 5 is a main thread of the GUI 8 that is an application. The thread 5, the thread 2, and the thread 3 are generated and destroyed, and the thread 1, the thread 2, and the thread 5 are generated according to a programmed algorithm or a user operation. The thread 3 is controlled.

  In the thread 1 which is the transmission processing of the RTP / RTCP packet, the user's voice is captured from the microphone and the PCM data is received (Capture). If necessary, the BGM synthesis unit 15 or the like captures the captured sample frame. The sound effects and BGM and sound are synthesized (Effect or Mixing), and the codec 13 compresses and encodes the data (encode). At this time, as described above, the codec 13 determines the compression encoding method based on the information for controlling the compression rate of the control information included in the RTCP packet received from another VoIP client. Then, the RTP / RTCP control unit 12 converts the compression-encoded data into RTP packets, and the VoIP call control unit 7 generates control information such as the encoding band and the number of subband divisions to be compressed by other VoIP clients. RTCP packetized (packet), these packets are transmitted (send).

  In addition to the RTP packet transmission process, an RTCP packet transmission process that stores control information for controlling the codec compression encoding method in the communication target communication device is performed.

  On the other hand, in the RTP packet receiving process, a fluctuation absorbing buffer provided before and after the decoding process provided on the receiving side of the VoIP client, that is, a fluctuation absorbing buffer (Forward Jitter buffer) BF1 for storing the encoded byte stream PCM, and a decoding A PCM buffer (Backward Jitter buffer) BF2 for storing data that has been converted into an uncompressed linear PCM is used.

  In the RTP / RTCP packet reception function, in the thread 2, the RTP / RTCP control unit 12 receives the RTP packet (Receive), and detects a lost packet such as a data loss occurring on the network and a data error of the transfer packet. Compensation processing is performed (parse), and the data is stored (push & pop) in the fluctuation absorbing buffer BF1 provided before the decoding processing. This is decoded and expanded by the codec 13 (decode) and stored as uncompressed data in a fluctuation absorbing buffer (PCM buffer) BF2 provided after the decoding process (push).

  In the thread 3, the USB camera driver 2a, the USB audio driver 2b, and the various drivers 2c read unpopulated data from the fluctuation absorbing buffer BF2 (pop) and reproduce the read data (sound device).

  Further, the effect effect and the synthesis (Effect or Mixing) of BGM and sound performed in the thread 1 may be performed after the data is read from the fluctuation absorbing buffer BF2 in the thread 3 and before the reproduction. Alternatively, a PCM buffer for storing uncompressed data may not be prepared, and data decoded by reading data from the fluctuation absorbing buffer may be reproduced without being stored in the fluctuation absorbing buffer BF2.

  Further, in addition to the RTP packet reception process, the VoIP call control unit 7 controls the compression encoding method in the codec 13, so that the RTCP packet reception process such as extracting control information included in the received RTCP packet is performed.

  Next, the codec change control method for real-time acoustic data stored in the RTP packet will be described in more detail. In the VoIP system according to the present embodiment, the compression rate of transmission data obtained by synthesizing BGM, sound effects, and the like with voice data to be transmitted, that is, the transmission rate of the data differs depending on the network congestion state without re-call control. By doing so, communication can be continued even for transmission data having high sound quality and a large amount of information.

  Specifically, for example, when the network is congested for some reason, the bit rate can be lowered by changing the codec (compression encoding method) without performing the recall control.

  For this reason, the RTP / RTCP control unit 12 always or periodically measures the network transmission rate from the received RTP packet. In accordance with the measurement result of the transmission rate, the VoIP call control unit 7 makes a request for retransmission of the missing packet in the RTCP packet, and simultaneously notifies the communication component of the frequency component to be compressed.

  Specifically, for example, when the transmission rate is reduced due to an increase in transmission delay (temporal delay) of the RTP packet and the VoIP call control unit 7 determines that the network is congested, RTP / RCTP The control unit 12 performs control so as to generate an RTCP packet for notifying that, for example, a high frequency component is deleted or compressed by reducing bit allocation as a frequency to be compressed in the transmission sound.

  Thereby, for example, in another VoIP client that has received an RTCP packet having an instruction (control information) to lower the bit rate, the control information is transmitted from the RTP / RCTP control unit of the other VoIP client via the VoIP call control unit. Is input to the codec, and the codec changes the compression encoding method. In this way, the transmission data in which the compression coding method is changed by the codec and the transmission rate is reduced is converted from the RTP / RCTP control unit 12 into an RTP packet and transmitted from another VoIP client. Thus, the VoIP client transmits the control information for controlling the transmission rate of the RTP packet transmitted from the other VoIP client as the other party by the RTCP packet, and controls the transmission rate of the RTP packet transmitted from the other VoIP client. can do.

  In this way, in response to a request from the VoIP client on the receiving side, another VoIP client on the transmitting side changes the compression encoding method, so that synthesized sound source data in which high-quality sound, BGM, and sound effects are synthesized, In other words, if the network bandwidth suddenly becomes crowded for some reason during data communication that needs to be transmitted at a high bit rate, it will continue without interrupting the conversation (communication) even if the sound quality temporarily decreases Can be made. That is, since the compression encoding method can be changed using the RTCP packet without using a call control message such as SIP, the call is not blocked.

  Further, the RTCP packet in the present embodiment is an RTCP APP packet that can be expanded as will be described later, and the RTCP packet can be used to make a retransmission request for the missing packet, and the retransmission request for the missing packet is made. At the same time, the frequency components to be compressed can be notified by other VoIP clients, so the processing becomes quick.

  Next, the VoIP system in the present embodiment will be described in more detail. In the above RTC / RTCP packet transmission / reception between VoIP clients, first, one VoIP client performs call signaling to the other VoIP client, and both recognize the IP address, port number, voice and acoustic compression method of the other party. There is a need to. Here, a PC (Personal Computer) having a VoIP client application 1 as shown in FIG. 3 is a UA (User Agent), and a call control method when VoIP is performed between the UAs will be described.

  FIG. 7 is a diagram showing a flow of message data exchanged when a call is made using the VoIP software 1 shown in FIG.

  UA1 and UA2 shown in FIG. 7 are SIPUAs that perform VoIP call control by SIP. In such VoIP communication, when operating as a start side (calling side) using SIP, it becomes UAC (UA Client), and when operating as a communication receiver side (calling side), UAS ( UA Server), and data (messages) are exchanged between both VoIP clients 111 and 121. The SIP message includes a request that is a message sent from the UAC to the UAS and a response that is a message returned from the UAS to the UAC. The request message body includes a session based on SDP (Session Description Protocol). Information is stored. Further, the predetermined response such as an OK response includes an SDP message for selecting a codec used for a call.

  First, INVITE SDP1 (D1) is sent from the calling side UA1 to the called side UA2 as a session participation request. When this arrives at the callee UA1, a request is received from the callee UA2 and a 100 Tryning response and a 180 Ringing response (D2) are sent indicating that the request is being processed. A 200 OKSDP response (D3) is sent indicating that it has been accepted. The 180 Ringing response indicates a state in which a ringing tone is sounded by the called side UA2, and a 200 OKSDP response is sent when the handset is picked up by the called side UA2. Then, an ACK (D4) is sent from the calling side UA1, thereby opening a port for communication. In this way, an RTP / RTCP session is established, and voice and acoustic data are transmitted and received (D5).

  On the other hand, when UA2 that is the called side becomes the calling side and a session participation request INVITE SDP2 (D6) is sent to UA1, the calling side UA2 (client side) When a 405 Not Acceptable response (D7) indicating an error is transmitted, the calling side UA2 that has received this transmits ACK (D8), and again sends a session participation request INVITE SDP2 (D9) to the called side UA1. resend.

  If the called side UA1 can normally receive this, a 200 OK response (D10) is sent to notify this, and an ACK (D11) is sent from the calling side, so that a new RTP / RTCP session is established. And transmit and receive acoustic data (D12).

  As described above, first, after performing call control, an RTP / RTCP session is established, and voice and acoustic data are transmitted and received.

  FIG. 8 is a diagram showing a description example of the SDP used in the call control shown in FIG. As shown in FIG. 8, among tags (annotation symbols) indicated by SDP, “v” indicates a version number, “o” indicates a session creator and information for session identification, and o = <user name> <SDP message identification number> <message identification subnumber> <network type> <address type> <address> “S” indicates the title of the session, “c” indicates connection data, and is described as s = <start time> <end time>.

  “M” indicates a media name and transport address, m = <media type> <port number> <transport protocol> <payload type>, “a” indicates a media attribute, m = Rtpmap: <payload type> <coding rule> / <clock>.

  In this way, the SIP can describe the IP address, port number, voice and acoustic compression method, and this SDP is mutually transmitted and received during call control. In the VoIP system, an RTP / RTCP session is established by this SIP and RTP / RTCP packets are transmitted / received. As described above, if the voice and acoustic compression method is established by this SIP call control, In order to make the transmission bit rate variable according to the degree of bandwidth congestion, it is necessary to perform the recall control again.

  Therefore, in the present embodiment, the arrival state of data on the receiving side, that is, the transmission rate on the transmitting side, is constantly or periodically measured using an APP packet that is an extension of the RTCP packet, and based on this transmission rate. In this case, it is determined whether to increase or decrease the transmission rate, and this is notified to the transmission side by an APP packet to control the transmission rate of data to be transmitted.

  Next, RTP / RTCP packets transmitted / received after establishing an RTP / RTCP session will be described. FIGS. 9A and 9B are diagrams showing the configuration of the RTP packet and the format of the header. FIGS. 10A to 10C show the configuration of the RTCP APP packet that can be expanded by the application and its header. It is a figure which shows the example of a format of (RTCP APP Application-defined Header) and a transmission report.

  RTP is a transport protocol for transmitting / receiving voice and moving images in real time in an IP network such as the Internet, and is recommended by RFC1889. RTP is located in the transport layer and is generally used with RTCP over UDP.

  And as shown to Fig.9 (a), an RTP packet consists of an IP header, a UDP header, an RTP header, and RTP data. In the RTP header, as shown in FIG. 9B, from the top, a version information storage unit (V: version, for example, V = 2) F1, a padding storage unit (P: padding) F2, an extension bit storage unit (X: extension) F3, CSRC (contributing source) count storage unit (CC) F4, marker information (M: marker) storage unit F4, marker bit storage unit (M: maker) F5, payload type information storage unit (PT : Payload type) F6, sequence number information storage unit (sequence number) F7, time stamp storage unit (time stamp) F8, SSRC identifier storage unit (synchronization source identifier) F9, and CSRC identifier storage unit F10 are provided to store CSRC identifiers. Real-time acoustic data is added behind the part F10.

  The version information storage unit F1 stores information indicating the RTP version. For example, when indicating RTP2, the version information indicating that is stored.

  The count storage unit F4 indicates the number of CSRCs (contribution source identifiers) indicated in the header.

  The payload type information storage unit F6 stores information indicating the type of real-time acoustic data, for example, information indicating video or audio.

  The sequence number information storage unit F7 counts up every time an RTP packet is transmitted / received in an RTP session, and stores a sequence number for recognizing the order of RTP packets to be transmitted / received.

  The time stamp storage unit F8 stores time stamp information related to the date and time when real-time acoustic data is created and updated.

  The SSRC identifier storage unit F9 and the CSRC identifier storage unit F10 store information for identifying the source on the data transmission side in the RTP session. The SSRC identifier is a synchronous transmission source identifier, is an identifier assigned so that a plurality of streams to be handled in combination by the same user share the same value, and the CSRC identifier is a contributing transmission source identifier, indicating a stream source . This is used when a plurality of streams are mixed and provided as one stream data.

  When transmitting real-time acoustic data according to RTP, the RTP / RTCP control unit 8 shown in FIG. 3 stores various types of information in the storage units and recognizes the various types of information stored in the storage units. Process to extract acoustic data.

  While RTP is a protocol that transmits / receives audio / video data itself, RTCP periodically evaluates line quality such as packet loss, delay jitter, round trip, etc., and matches the bandwidth. A protocol for transmitting / receiving information in order to realize real-time communication.

  By using this RTCP, it is possible to perform dynamic processing such as changing the transmission rate by estimating the network state or the like based on information fed back from the other party. In addition, since information indicating who is currently sending data and who is receiving it is also sent at the same time in the RTCP packet, it is possible to know the current participant information.

  As shown in FIG. 10A, the RTCP packet includes an IP header, a UDP header, an RTCP header, and RTCP data. In the header of the RTCP APP packet, which is an expandable RTCP packet, as shown in FIG. 10B, the version information storage unit (V: version, for example, V = 2) F11, the padding storage unit ( P: padding) F12, subtype storage unit F13, packet type (PT) storage unit F14, report length storage unit (length) F16, SSRC / CSRC identifier storage unit F17, ASCII (American Standard Code for Information), A Name storage unit F18 described in (American standard code for information exchange) is provided, and a data storage unit (Application-Dependent Data) F19 for storing application-specific data is added thereto.

  The type of RTCP packet is recorded in the packet type PT storage unit F14, and in this embodiment, this packet type PT = APP (Application: application specific information) = 204 (packet type value) is described. APP indicates that it is a packet for notifying application-specific control information outside the RTCP standard.

  As described above, the RTP / RTCP control unit 8 shown in FIG. 3 is a RTCP APP packet for receiving a retransmission request for a lost packet and an RTP packet used for absorbing jitter when the RTP packet is received. The size and the like of the fluctuation absorbing buffer can be described in the data storage unit F19 and transmitted. At the same time, according to an instruction from the VoIP call control unit 3, it is possible to describe and transmit information on frequency components to be compressed in transmission data to be transmitted by the communication partner.

  Here, the RTCP packet includes an RTCP SR (Sender Report) packet of a type sent from a sender of RTP data and an RTCP packet RTCP RR (Receiver Report) packet of a type sent from a receiver of RTP data.

  The SR packet is transmitted from a terminal that is transmitting a stream to another terminal. Transmission information (sender info) that is information about a stream transmitted by the own device and reception of the stream for each received stream Including a report block (reception report block) for reporting the state (packet discard rate, jitter, etc.) to the transmitting device, and the RR packet is for notifying information on a stream received from another communication device, Similarly, each received stream includes a report block for reporting the reception status of the stream to the transmitting apparatus.

  As shown in FIG. 10C, this report block includes a synchronization source (SSRC) identifier of a packet sender, an RTP loss rate, a number of lost RTP packets, a reception sequence number, and jitter of an arrival time interval. The average value, the last received SR transmission time (LSR: Last SR timestamp), and the time from the last SR received time to sending this RR (DLSR: Delay since Last SR) are entered.

  Therefore, the transmission side manages the number of transmission RTP packets and the number of transmission RTP bytes when transmitting RTP data, and the reception side receives the number of received RTP packets when receiving RTP data. Management information such as the number of lost RTP packets and jitter of arrival time is managed.

  FIG. 11 is a diagram showing messages exchanged when exchanging RTCP APP packets in the present embodiment. In an RTCP packet including an SR block and an RTCP packet including an RR block exchanged between UA1 and UA2 which are both RTP Sender and Receiver, as shown in FIG. Bandwidth), number of subband division blocks (sub band Numbers), fluctuation absorption buffer size before and after decoding processing unit (convertible to time) (Forward and Backward Jitter Buffer size), data size queued in the absorption buffer ( (Can be converted into time) (Buffer queued size) and RTP packet retransmission request information (Re-Request: sequence number) can be described. The sequence number is used for the RTP packet retransmission request.

  The UA1 transmits a transmission sound source stream in which BGM, sound effects, and the like are synthesized with the voice of the user of the UA1, using an RTP packet. The UA 2 also transmits a synthesized sound obtained by synthesizing BGM, sound effects, and the like with the user's voice of the UA 2 using an RTP packet.

  For example, UA1 adds to the RTCP APP packet a report block in which UA1 as the stream receiving device reports the stream reception status to UA2 as the stream transmitting device, and also according to the arrival status of the RTP packet sent from UA2. Thus, the encoding bandwidth and the number of subband division blocks as control information are described. Receiving this, UA2 changes the transmission rate by changing the encoding band and the number of subband division blocks of transmission data to be transmitted to UA1, based on the encoding bandwidth and the number of subband division blocks.

  Similarly, the UA2 can notify the UA1 of the encoding bandwidth and the number of subband division blocks as control information for controlling the transmission rate according to the arrival state of the RTP packet sent from the UA1. In response, the UA 1 changes the band to be encoded and the number of subband division blocks, and changes the transmission rate.

  Furthermore, a time from when an RTCP packet including an SR (Sender Report) block is transmitted between the UA1 and the UA2 to when an RTCP packet including an SR (Receiver Report) block is received is an RTT (Round Trip Time). UA1 notifies the sequence number of the missing packet that can be retransmitted from UA2 according to this RTT and the fluctuation absorbing buffer size before and after its own decoding processing unit. That is, based on the RTT calculation result, the time information until the retransmission requested packet is sent is obtained, and the packet decoding start scheduled time or reproduction scheduled time is later than the time when the retransmission requested packet arrives Can make a retransmission request.

  The application specific information APP can also describe information such as the fluctuation absorbing buffer size before and after its own decoding process, the amount of data occupying the fluctuation absorbing buffer, and the other party receiving the notification. The VoIP client may change the transmission rate of data to be transmitted based on these values.

  In this way, in UA1 and UA2, the frequency component of the sound to be compressed in the RTP packet sent from the communication partner can be notified to the call partner by the encoding bandwidth and the number of subband division blocks included in the RTCP APP packet. Even if the network bandwidth suddenly becomes congested for some reason during a high-quality voice call, the conversation can be continued without being cut off by lowering the transmission rate.

  That is, in this embodiment, for example, the UA2 on the receiving side that has received the RTP packet storing the transmission data from the UA1 transmits an RTCP APP packet that requests the UA1 to change the transmission rate based on the reception result. . Receiving this, UA1 retransmits the missing packet to UA2 based on the control information included in the received APP packet, and changes the compression encoding method of the transmission data based on the information of the frequency component to be compressed. The transmission rate of data to be transmitted can be made variable. In this way, the VoIP client on the transmitting side that transmits the RTP packet transmits the data with the codec changed as appropriate in response to the request on the receiving side of the RTP packet, thereby eliminating the need for recall control to notify the codec change. Communication is not interrupted.

  Furthermore, in this embodiment, the transmission rate is measured at the reception side constantly or periodically, and the codec on the transmission side is automatically controlled according to the transmission rate. The same processing can be performed based on the user operation.

  That is, as shown in the GUI 8 shown in FIG. 4, for example, an operation unit such as a sound quality control unit 69 for increasing or decreasing the transmission rate is provided, and a high-quality sound in which only a voice call or BGM is added to the voice is provided. The user can instruct the transmission rate according to whether the call is based on the data, or if the user feels that it is difficult to hear, for example, voice and sound, the user operates the operation unit to set the transmission rate. Can be changed. In addition, an operation unit that can select whether to cut high-frequency components or acoustic data when lowering the transmission rate is provided, and the VoIP call control unit 3 uses the RTCP APP packet based on a user operation. The transmission rate on the sender side may be controlled. Conversely, the sound quality may be improved by increasing the transmission rate according to the user's request.

  Next, a description will be given of how to obtain an RTT that indicates the time required for a message to reciprocate after a packet is transmitted. As described above, this RTT is referred to by the VoIP call control unit 3 together with the size of the RTP packet reception buffer used to absorb the jitter when the RTP packet is received when the retransmission request of the lost packet is made. It is what is done.

  FIG. 12 is a sequence diagram showing an RTT calculation method using RTCP. As shown in FIG. 12, when an RTCP SR packet is first sent from UA1 between UA1 and UA2, which are both RTP Sender and Receiver, an RTCP RR packet is sent from UA2. In the RTCP RR packet, LSR and DLSR are described as report blocks.

  The LSR is an NTP (Network Time Protocol) time stamp of the last received SR packet. For example, in FIG. 12, the LSR is a time (10 Nov 1995 11: 33: 25.125) when the UA 1 transmits the RTCP SR packet. The UA 1 that has received the RR packet calculates the RTT (= LSR-A) from the arrival time A (10 Nov 1995 11: 33: 36.5) of the RTCP RR packet and the LSR described in the report block of the RR packet. Can do.

  The RTT includes a DLSR or the like indicating a value representing the elapsed time until the report block is transmitted when the SR packet is last received in units of 1/65536 seconds. This changes depending on the transmission rate and the degree of network congestion.

  In this way, the UA 1 estimates the network load from the calculated RTT, packet arrival time, and the like, requests retransmission of the missing packet, and adjusts the transmission rate of the stream. This RTT may be calculated every time an RTCP packet is transmitted / received, or may be calculated every elapse of a predetermined time or may be calculated when the codec is changed.

  Next, a specific example of a high-efficiency acoustic compression encoding and decoding expansion method using auditory psychological characteristics will be described. 13 and 14 are block diagrams showing a high-efficiency acoustic compression encoding unit and a high-efficiency acoustic decoding / decompression unit, respectively. The high-efficiency acoustic compression encoding unit 70 corresponds to the encoder 31 shown in FIG. 2, and as shown in FIG. 13, a time-frequency decomposition unit 71, a quantization unit 72, an auditory psychological model unit 73, A band allocation unit 74 and a multiplex 75 are included.

  The time-frequency decomposition unit 71 blocks or frames the time-axis signal in predetermined time units, converts the time-axis signal for each frame into a signal on the frequency axis (spectrum conversion), and divides it into a plurality of frequency bands To do.

  The psychoacoustic model unit 73 generally performs processing such as dividing an audio signal into a plurality of bands (for example, 25 bands) with a bandwidth called a critical band (critical band) in which the bandwidth becomes wider as the frequency increases. The band allocation unit 74 performs predetermined bit allocation for each band or adaptive bit allocation (bit allocation) for each band. For example, when the coefficient data obtained by the modified discrete cosine transform (MDCT) process is encoded by bit allocation, the MDCT coefficient data for each band obtained by the MDCT process for each frame is used. On the other hand, an adaptive number of bits is assigned.

  Here, the encoding band and the number of subband division blocks included in the control information stored in the RTCP packet described above are supplied to the psychoacoustic model unit 73, for example, and the data to be encoded is only the data of the specified encoding band. Then, processing such as dividing this encoding band into the number of subband divided blocks is performed.

  The quantization unit 72 performs quantization by determining a quantization step or a quantization size based on the number of bits assigned for each band.

  The multiplex 75 multiplexes the quantized data together with sub-information such as the number of bits allocated by the band allocation unit and outputs the multiplexed data.

  As shown in FIG. 14, the high-efficiency acoustic decoding / decompression unit 80 includes a demultiplexer 81, an inverse quantization 82, and a time-frequency reconstructing unit 83. The demultiplex 81 receives high-efficiency encoded data and demultiplexes the data. The inverse quantization unit 82 inversely quantizes the quantized data based on the sub information such as the band information extracted from the output multiplex 81, and the time frequency reconstruction unit 83 converts the data on the time axis to the data on the frequency axis. Convert to and output.

  According to such high-efficiency acoustic compression coding, the total bit rate of all the acoustic information channels can be made variable, and bit allocation can be performed so as not to exceed a certain maximum value.

  That is, the frequency band to be compressed is selected according to the encoding band specified in the APP packet, and the encoding band is divided into the number of subband division blocks specified in the APP packet, so that it is included in the APP packet from the communication partner. The transmission bit rate can be changed by changing the compression rate of real-time audio data to be encoded by reducing or increasing the number of bits of output data based on the encoding band and the number of subband division blocks It can be.

  Here, the frequency component to be compressed is specified in the APP packet. However, for example, the transmission bit rate is specified, and the high-efficiency compression encoding unit 70 on the transmission side specifies the specified transmission bit. You may make it compression-encode so that it may become a rate. Furthermore, information such as the number of bits to be assigned to each band as control information and the quantization step or size to be quantized by the quantization unit may be specified.

It is a schematic diagram which shows an example of the VoIP system in embodiment of this invention. It is a block diagram which shows the principal part which consists of the VoIP client 111 and the internet 130 among the VoIP systems in embodiment of this invention. It is a figure which shows the VoIP client application at the time of using general PC as a VoIP client which the VoIP system in embodiment of this invention has. It is a schematic diagram which shows an example of GUI in the said VoIP client application. It is a block diagram which shows the hardware constitutions of the VoIP client (VoIP client) in the VoIP system in embodiment of this invention. It is a functional block diagram which shows the process which transmits / receives an RTP / RTCP packet among the VoIP client applications shown in FIG. It is a figure which shows the flow of the message data exchanged in the call control at the time of performing VoIP using the said VoIP client application. It is a figure which shows the example of description of SDP used in the case of call control. (A) And (b) is a figure which shows the structure of a RTP packet, and the format of a header, respectively. (A) thru | or (c) is a figure which shows the format of the structure of the RTCP APP packet expandable by an application, its header (RTCP APP Application-defined Header), and a transmission report. It is a figure which shows the message exchanged at the time of the exchange of RTCP APP message in the VoIP system in this Embodiment. It is a sequence diagram which shows the calculation method of Round Trip Time using RTCP. It is a block diagram which shows the highly efficient acoustic compression encoding part using the psychoacoustic characteristic. It is a block diagram which shows the highly efficient acoustic decoding expansion | extension part using the psychoacoustic characteristic.

Explanation of symbols

1 VoIP client application, 21 transmitting means, 22 microphone capture, 23 sound effect file reading unit, 24 BGM file reading unit, 25, 26, 31 decoder, 27, 28, 29 gain adjusting unit, 30 synthesis unit, 31 encoder, 32 Packetizing unit, 33 transmitting unit, 41 receiving means, 42 receiving unit, 43 depacketizing unit, 44 dejittering unit, 45 packet compensating unit, 46 decoder, 47 reading unit, 48 decoding unit, 49, 50, 51, 52 gain adjusting unit , 53 synthesis unit, 54 output unit, 55 speaker, 100 VoIP communication system, 111, 121 VoIP client, 110, 120 user, 112, 122 web browser, 134 web server, 113, 123 sound source data, 130 interface Tsu door

Claims (19)

In a communication device that transmits and receives real-time acoustic data via the Internet,
Compression encoding means for compressing and encoding real-time acoustic data including speech;
Data packet generation means for generating a data packet storing data compressed and encoded by the compression encoding means;
Control packet generating means for generating a control packet storing management information for managing at least transmission and reception of the data packet;
Transmitting means for transmitting the data packet and the control packet to one or more other communication devices;
Receiving means for receiving data packets and control packets from the other communication device;
Control means for generating control information for controlling the transmission rate of the data packet from the other communication device,
The communication device, wherein the control packet generation unit generates the control packet storing the control information together with the management information. The call device according to claim 1, wherein the control means measures a transmission rate of a data packet from the other call device received by the receiving means, and generates the control information based on the measurement result. . The call device according to claim 1, wherein the control means generates the control information based on a user operation. The call device according to claim 1, wherein the control information includes information for controlling a compression rate of real-time acoustic data stored in a data packet from the other call device. The communication device according to claim 4, wherein the information for controlling the compression rate includes information for designating a frequency band to be compressed in real-time acoustic data stored in a data packet from the other communication device. . The communication device according to claim 4, wherein the information for controlling the compression rate includes information for designating a subband division number of real-time acoustic data stored in a data packet from the other communication device. The reception means has a reception buffer for buffering the received data packet,
The control means generates retransmission request instruction information for instructing a retransmission request for a missing data packet according to the size of the reception buffer,
The call device according to claim 1, wherein the control packet generation unit generates the control packet storing the retransmission request instruction information. The call device according to claim 1, wherein the real-time acoustic data stored in the data packet is a combination of voice and background music and / or sound effects. The compression encoding means changes a compression rate of real-time acoustic data to be transmitted to the other call device based on control information for controlling the transmission rate stored in a control packet from the other call device. The call device according to claim 1, wherein: In a call method that transmits and receives real-time acoustic data via the Internet,
A compression encoding process for compressing and encoding real-time acoustic data including speech;
A data packet generating step for generating a data packet storing the data compressed and encoded in the compression encoding step;
A control packet generation step of generating a control packet storing management information for managing at least transmission and reception of the data packet;
A transmission step of transmitting the data packet and the control packet to one or more other communication devices;
A receiving step of receiving a data packet and a control packet from the other communication device;
A control step of generating control information for controlling the transmission rate of data packets from the other communication device,
In the control packet generation step, the control packet storing the control information together with the management information is generated. 11. The call according to claim 10, wherein in the control step, a transmission rate of a data packet from the other call device received in the reception step is measured, and the control information is generated based on the measurement result. Method. The call method according to claim 10, wherein in the control step, the control information is generated based on a user operation. The call method according to claim 10, wherein the control information includes information for controlling a compression rate of real-time acoustic data stored in a data packet from the other call device. The call method according to claim 13, wherein the information for controlling the compression rate includes information for designating a frequency band to be compressed in real-time acoustic data stored in a data packet from the other call device. . The call method according to claim 13, wherein the information for controlling the compression rate includes information for designating a subband division number of real-time acoustic data stored in a data packet from the other call device. In the control step, retransmission request instruction information for instructing a retransmission request for a missing data packet is generated according to a capacity of a reception buffer for buffering the received data packet,
The call method according to claim 10, wherein, in the control packet generation step, the control packet storing the retransmission request instruction information is generated. The call method according to claim 10, wherein the real-time acoustic data stored in the data packet is a combination of voice and background music and / or sound effects. In the compression encoding step, the compression rate of the real-time acoustic data transmitted to the other call device is changed based on the control information for controlling the transmission rate stored in the control packet from the other call device. The call method according to claim 10, wherein: In a call system in which two or more call devices transmit and receive real-time acoustic data via the Internet,
Each communication device
Compression encoding means for compressing and encoding real-time acoustic data including speech;
Data packet generation means for generating a data packet storing data compressed and encoded by the compression encoding means;
Control packet generating means for generating a control packet storing management information for managing at least transmission and reception of the data packet;
Transmitting means for transmitting the data packet and the control packet to one or more other communication devices;
Receiving means for receiving data packets and control packets from the other communication device;
Control means for generating control information for controlling the transmission rate of the data packet from the other communication device,
The control packet generation means of one of the at least two communication devices generates the control packet storing the control information for controlling the transmission rate of the data packet from the other communication device together with the management information,
The compression coding means of the other call device changes a compression rate of real-time acoustic data transmitted to the one call device based on the control information stored in a control packet from the one call device. Call system characterized by.

Images