JP2007150917A - Communication terminal and display method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a communication terminal and a display method thereof which can adaptively and optimally update the size and position of a display image area (screen) depending on conditions without requiring operation by a user, can make rearrangement (movement) of the display image area continuous, and can arrange it with the optimum size even if it has a different shape. <P>SOLUTION: When a screen exists on a designated position on an image surface, an encoder 20 as a transmission source generates corresponding designation information, screen information and volume information, and transmits the information to an opposite party in communication. A decoder 30 has a function of displaying a multi-screen, and functions of calculating a display magnification factor of the screen based on a line segment connecting the centers of the screen, the thickness of a reference shape and the volume of sounds. and optimally forming a plurality of screens on the image surface by controlling the movement and new generation of the screen based on this display magnification factor. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a communication terminal such as a mobile phone and a display method thereof, and more particularly to a communication terminal capable of multipoint communication and a display method thereof.

  A representative example of multipoint communication is a video conference system. In the video conference system, a plurality of terminals are connected via an MCU (Multi-point Control Unit). The MCU divides and synthesizes image data sent from a large number of terminals on one screen, and transmits it to each terminal together with audio data, thereby realizing a video conference connecting multiple points.

Basically, when dividing and synthesizing the images of each site into one image,
(1) When one image is equally divided (for example, 4 divisions, 9 divisions),
(2) Taking one large image area and dividing the remaining area into equal parts;
(For example, 6 divisions).

In the case of (1), images from bases connected by MCU are synthesized using the same area.
In the case of (2), a large area is allocated to the talking base, and images from the remaining bases are assigned to the remaining equally divided areas to be combined.

  In any case, since the video conference system uses a large-screen monitor, even if the images of a plurality of locations are divided and combined into one image, the size of the image showing each location is sufficiently large. As long as you are alone, the face of that person will not be difficult to recognize. A video conference system is disclosed in, for example, Patent Documents 1 and 2 and the like.

1A to 1E are diagrams showing examples of display screens of a terminal such as a personal computer (PC) at the time of multilevel communication in a general video conference system.
In the example of FIG. 1, the screen 1 is divided into a predetermined frame (rectangular) window.
For example, the screen 1 is formed of one large window (square) 2 and a plurality of small windows (squares) 3-1 to 3-5, and displays a speaker on the large window 2.
In this case, since the window size and the number of divisions are fixed and the captured image is displayed as it is, the size of the face varies depending on the shooting state.

  In the case of a general PC window control, it is possible to freely change the window size and select the window by dragging the window with the mouse.

By the way, mobile communication terminals such as mobile phones are not only for voice calls, but have become more sophisticated year by year, such as mail, web access, games, cameras, videophones, media players, radios, and televisions.
At present, a videophone in a mobile communication terminal is selected to connect by a videophone when a call is made.

  However, with the progress of support for packet communication, VoIP that supports packet communication is used as the voice call itself, and the camera is activated during the call and switched to voice and video calls, and vice versa. The main usage is to stop and make only voice calls. In addition, send documents (ex. Mail), address data, images (still images, moving images), audio, etc. saved on your terminal to the other party on the call, )) You can watch at the same time or watch the website at the same time.

As described above, when the mobile communication terminal is converted to IP due to high functionality, it is possible to communicate with a plurality of other parties (including a server) at the same time.
In this case, it is necessary to handle a plurality of screens with one terminal. As a method of handling a plurality of screens, there is a method of handling pages in a page turning manner (such as with a PDA).
Japanese Patent Laid-Open No. 06-141310 Japanese Patent Application Laid-Open No. 06-141311

By the way, for example, in a portable IP-TV phone, since the screen size is small, when a call with images is made by a plurality of people, the size of each person's face becomes small.
In order to be able to change the size when the user moves the window like a PC window, the screen size and operation keys are limited and difficult.
In addition, when multiple people talk at the same level, the screen is not compatible.

Furthermore, when the screen size is set according to the volume, there is a problem that a useless space is increased or the entire screen cannot be displayed in the screen.
Specifically, when the volume of all the screens is small, a small screen floats on the screen. On the other hand, when the volume of all screens is maximum, there is a problem that the total area of the display screen exceeds the screen area.
On the other hand, by locally normalizing the total area to the screen area, it is possible to fit within the screen, but the rearrangement (movement) of the screen position at the next time becomes discontinuous.

  It is an object of the present invention to adaptively and optimally display the size and position of the display image area (screen) according to the situation such as the volume level and the number of display image areas (screens) to be displayed without the user's operation. A communication terminal capable of updating the display image area, and rearranging (moving) the display image area continuously, so that even a different shape can be arranged in an optimum size, and a display method thereof It is in.

  A first aspect of the present invention is a communication terminal that reproduces received image data and audio data, a display unit that displays an image, and a plurality of images that are extracted from a specific area and displayed on the display unit A plurality of display areas for displaying each of the display area, calculating the display magnification of the display area based on at least the line segment connecting the center of the display area of the image, the thickness of the reference shape, the volume of the sound, Control means for controlling the movement and new generation of the display area based on the display magnification to form a plurality of display areas on the display screen of the display means.

  Preferably, the display area is associated with a center position coordinate (P (i)) indicating a display position of the display area, a reference shape (Unit (i)) indicating the shape of the display area, and the display area. The sound is loud (V (i)), and the display magnification (R (i)) when the display area is displayed on the screen, the control means as the display magnification (R (i)) , The line segment (L (i, j)) connecting to the center position coordinates (P (j)) of the surrounding screen, and the thicknesses (Lm (i, j), Lm (j, i)) and the provisional display magnification (R (i, j)) calculated based on the sound volume (V (i), V (j)).

  Preferably, the control means sets the audio volume (V (k) = 0) and thickness (Lm (k, i) = 0) at a point perpendicular to the screen boundary from the center of the display area. Set and calculate the display magnification (R (i, k)).

  Preferably, the control means moves the display area to a position where the display magnification (R (i)) is maximized.

  The control means generates the center of the new display area at the position where the display magnification (R (k)) is the largest.

  Preferably, the control means equalizes the area of the reference shape.

  Preferably, the control means draws a separation line between the screens formed in a reference shape, and sets the area separated by the separation line as a new display area.

  Preferably, the information on the image to be displayed is included in the reception information from the transmission side device, and the control unit calculates display magnification based on the reception information, and controls movement of the display area and new generation. To do.

  A second aspect of the present invention is a display method of a communication terminal that reproduces received image data and audio data, and a line segment connecting the centers of display areas of a plurality of images to be extracted and displayed in a specific area. The display area display magnification is calculated based on the thickness of the reference shape and the volume of the sound, and a plurality of displays are displayed on the display screen by controlling the movement and new generation of the display area based on the display magnification. An area is formed and a plurality of display areas including an image to be displayed are displayed.

  According to the present invention, the size and position of the display image area (screen) are adaptively and optimally according to the situation such as the volume level and the number of display image areas (screens) to be displayed without the user's operation. In addition, the display image area can be rearranged (moved) continuously, and even with different shapes, it can be arranged in an optimum size.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  2 and 3 are diagrams illustrating a configuration example of the mobile communication terminal according to the embodiment of the present invention. FIG. 2 is a block diagram illustrating an encoding apparatus, and FIG. 3 is a block diagram illustrating a decoding apparatus. .

  The mobile communication terminal 10 includes an encoding device 20 serving as a transmission source and a decoding device 30 serving as a reception side, and is configured to be capable of multipoint communication.

The encoding device 20 has a function of transmitting the encoded audio data and image data to the network as a packet by adding instruction information for the receiving terminal, image top and bottom information, and the like.
The instruction information of the transmission source added to the audio data and the image data includes information for identifying the transmission source of the instructed image (for example, IP address, MAC address) and position information indicating the position on the received image. Including.
The encoding device 20 serving as a transmission source generates corresponding instruction information, screen information, and volume information when the transmission source has a screen (the screen will be described in detail later) at the indicated position on the screen. It has a function to send to the other party in communication.

  The encoding apparatus 20 in FIG. 2 encodes a voice input unit 201 composed of a microphone or the like, an image input unit 202 such as a digital camera, an operation unit 203 capable of key input or the like, and voice data input by the voice input unit 201. The audio encoding processing unit 204, the image encoding processing unit 205 that encodes the image data input from the image input unit 202 and cut out into a predetermined area, and receives the top and bottom of the captured image based on the top and bottom information associated with the captured image Detected by a top / bottom correction unit 206 that corrects the screen to match the top / bottom of the display screen (terminal screen), a face area detection unit 207 that detects and extracts a face area from the captured image, and a face area detection unit 207. A screen discriminating unit 208 that discriminates a screen (display image area to be displayed) to be used based on the face area and generates screen information. Based on the input information of the operation unit 203 and the input volume measuring unit 210 that generates the volume information by measuring the input volume by the voice input unit 201 by cutting out the corresponding area from the received image based on the determination of the lean determination unit 208. A terminal control unit 211 that controls the terminal, a control information generation unit 212 that generates control information including instruction information, top and bottom information, screen information, volume information, and the like based on instructions from the terminal control unit 211, and stores images and videos A storage unit 213, encoded audio data and image data, control information, a transmission packet generation unit 214 that generates image / video data read from the storage unit 213 based on an instruction from the terminal control unit 211 as a transmission packet; In addition, the transmission packet generated so that wireless communication with the network can be performed via the network is performed on the other party's terminal or A network interface (I / F) 215 for transmitting to the server.

The decoding device 30 has a function of reproducing audio data and image data transmitted from the encoding device 20 of the communication partner (transmission source) and received via the network.
For example, when performing multipoint communication, the decoding device 30 selects a screen (a display area whose size is controlled) to use an image including a face as a specific area based on the control information of the received image. It has a function to display and emit sound.
The decoding device 30 has a function to divide in a circular (concept including an ellipse) window from which a dead zone is eliminated when the screen is displayed.
The reason for the division into the circular (elliptical) windows is as follows.
In general, the screen is divided into rectangles. The human face is basically oval, and the four corners of the rectangle are dead zones. This dead zone results in a narrow (small) area for displaying the face.
Therefore, in this embodiment, it is configured so as to be divided by a circular (elliptical) window from which this dead zone is eliminated.
Further, the decoding device 30 has a function of displaying a multi-screen, calculates a screen display magnification based on a line segment connecting the centers of the screens, a thickness of a reference shape, and a loudness of the sound. By controlling the movement and new generation of the screen based on the above, it has a function of optimally forming a plurality of screens on the screen.
Specific processing will be described in detail later in association with the drawings.

  The decoding device 30 of FIG. 3 is a network interface that receives packets including audio data, image (video) data, control information and instruction information, screen information, volume information, and the like that are wirelessly communicable with a network and transmitted from a transmission source. (I / F) 301, operation unit 302 capable of key input, etc., and packet received by network interface 301 are analyzed, and voice data, image data, transmission source address, control information (top and bottom information, instruction information, etc.) are extracted. The received packet analysis unit 303, the audio decoding processing unit 304 that decodes the audio data extracted by the reception packet analysis unit 303, the video decoding processing unit 305 that decodes the video data extracted by the reception packet analysis unit 303, and the video decoding process. Video data decoded by the unit 305, transmission source address, control information, screen information Display image control unit 306 that controls the size and display form of a screen (display window) to be displayed based on the information, size information, and top and bottom information, and a volume correction unit that corrects the volume of the audio decoded by the audio decoding processing unit 304 307, an audio output unit 308 such as a speaker that produces sound with the volume corrected by the volume correction unit 307, an image correction unit 309 that corrects an image whose size and display form are controlled by the display image control unit 306, and an image correction unit 309. A display unit (image output unit) 310 such as an LCD for displaying an image via the terminal, and a local terminal control unit 311 for giving control information (top and bottom information) to the display image control unit 306 based on input information from the operation unit 302 .

  Note that the encoding device 20 and the decoding device 30 can share the operation units 203 and 302, the network interfaces 210 and 301, the terminal control unit 211, and the own terminal control unit 311.

  Hereinafter, a more specific configuration and function of the display image control unit 306, which is a characteristic part of the present embodiment, and a specific configuration and display mode example of the screen will be described in order.

  The display image control unit 306 in FIG. 3 is based on the control information analysis unit 3061 that extracts screen information, size information, top and bottom information, and instruction information based on the control information supplied from the received packet analysis unit 303, and based on the screen information. A masking processing unit 3062 that masks the video decoded by the video decoding processing unit 305, a display magnification calculating unit 3063 that calculates a display magnification of a screen (display image area) to be displayed based on the size information, and a display magnification calculation A reduction / enlargement processing unit 3064 for reducing / enlarging the image after the masking process according to the display magnification calculated by the unit 3063, and a display position calculation unit for calculating a display position according to the display magnification and the top / bottom information calculated by the display magnification calculation unit 3063 3065 and the position on the display unit 310 obtained by the display position calculation unit 3065 Having a mapping processing unit 3066 for mapping the image obtained by small-enlargement processing section 3064.

  The screen whose size and display form are controlled by the display image control unit 306 of this embodiment is displayed as a multi-screen that displays a plurality of screens on one screen.

In the display magnification calculation unit 3063 of this embodiment, the screen is associated with the center position coordinate (P (i)) indicating the display position of the screen, the reference shape (Unit (i)) indicating the shape of the screen, and the screen. Loud (V (i)), and display magnification (R (i)) when displaying the screen on the screen, the display magnification (R (i)) is the center position coordinates of the surrounding screen ( P (j)) and the line segment (L (i, j)), the thickness of the reference shape on the line segment (Lm (i, j), Lm (j, i)), and the volume of the voice The smallest value among the temporary display magnifications (R (i, j)) calculated based on (V (i), V (j)).
In the display magnification calculator 3063, the audio volume (V (k) = 0) and thickness (Lm (k, i) = 0) are set at the point that is perpendicular to the screen boundary from the center of the screen and displayed. The magnification (R (i, k)) is calculated.
Further, the screen moves to a position where the display magnification (R (i)) is maximized.
The screen generates the center of the new screen at the position where the display magnification (R (k)) is the largest.
The reference shape has the same area.
Furthermore, a separation line is drawn between the screens formed in the reference shape, and an area separated by the separation line is set as a new screen.

  Next, the calculation of the display magnification of the screen whose size and display form are controlled by the display image control unit 306 according to the present embodiment, the calculation of the generation position of the new screen, the calculation of the movement position of the screen, etc. will be described more specifically. To do.

  As shown in FIG. 4, each screen 40 has a reference shape (Unit). The screen 40 on the screen of the display unit 310 displays the reference shape (Unit) enlarged or reduced according to the display magnification (R).

Calculation of display magnification (R) :
The display magnification calculator 3063 has a distance (L (i, j)) between the centers of the screen i and the screen j and a thickness (Lm (i, j) in the reference shape (Unit) from the center of each screen to the direction. ), Lm (j, i)), and based on the received audio volume (V (i), V (j)) in the content displayed on each screen, from screen j on screen i The calculated display magnification (R (i, j)) is calculated as follows.

  The display magnification between the surrounding screens is calculated, and the smallest value among the display magnifications is set as the actual display magnification (R (i)) as shown in the following equation.

Calculation of new screen generation position :
The display magnification calculator 3063 places a temporary center on the screen and calculates the display magnification (Rmin) at each center. Of each display magnification (R), the position having the largest value is set as the center position for generating a new screen.

  The center (P (k)) satisfying this condition is set as the center position of the new screen.

Calculation of screen movement position :
Each screen calculates the display magnification (R) at each position within a certain distance (set I) from the current (t) position, and the position having the largest value among the display magnifications is calculated at the next time (t + Δt ).

  Move to the center (P (t + Δt)) that satisfies this condition.

  The screen position moves on the screen over time. For this reason, in the generation of a new screen, it is not necessary to perform calculation for all empty positions on the screen. That is, even if a new generation position is determined for some points on the screen and the positions are arranged based on the result, the display magnification moves to the position where the display magnification is the highest as time passes. Thereby, it becomes possible to reduce the calculation load in generation.

Since the positional relationship of the screen changes at any time, it is necessary to calculate the thickness (Lm (*)) of the reference shape in the display magnification (R (*)) calculation with respect to the direction at that time.
The thickness can be calculated by drawing a digital straight line from the center to the target direction (for a complicated shape). However, this increases the calculation load. For this, by referring to a table in which the thickness for each angle is calculated in advance for each reference shape, it is possible to reduce the calculation load when calculating the display magnification.

Wall treatment on all sides of the screen :
Each screen 40 calculates the display magnification (R) according to the following calculation rule between the four walls.
As shown in FIG. 5, the point dropped vertically from the center of the screen to the wall is taken as the center of the calculated wall, and the line segment between the centers (L (i, k)) and the thickness in the reference shape (Unit) ( Lm (i, k), Lm (k, i)) and the magnitude (V (i), V (k)) of the voice received by the screen are calculated. At this time, the sound volume on the wall is calculated as (V (k) = 0) and the thickness of the reference shape (Lm (k, i) = 0) in the same manner as the calculation of the display magnification (R) described above. I do.
When calculating the display magnification (R (i)) for each screen 40, the display magnification with the wall (R (i, k)) is the same as the display magnification with the surrounding screen (R (i, j)). And the display magnification (R (i)) when actually displaying the smallest value among these

An example in which the ratio of the sound volume (V (0), V (1)) is changed on the screen (S (0), S (1)) whose reference shape is an ellipse is shown in FIGS. Shown in (C).
6A to 6C, from the left, the audio volume ratio (V (0): V (1)) is 1: 1, 2: 1, 3: 1. Thus, the screen size can be adaptively changed according to the volume.

7A to 7 (A) to 7 (A) to 7 (A) to 7 (A) to 7 (A) to 7 (A) to 7 (A) to (E) in which the number of screens formed on the screen is increased or decreased. C).
7A to 7C, the numbers of screens are 2, 3, and 4 from the left.
In this way, it is possible to adaptively change the screen size according to the number of screens and form all the screens in the screen.

In a screen (S (0), S (1), S (2), S (3)) with an elliptical reference shape, the number of screens formed on the screen is increased or decreased, and the sound of one of these screens is recorded. FIGS. 8A to 8C show examples in which the size is double the size of the sound of other screens.
8A to 8C, the number of screens is 2, 3, and 4 from the left, and the audio volume ratio (V (0): V (1)) is 2: 1. (V (0): V (1): V (2)) is 2: 1: 1, ratio (V (0): V (1): V (2): V (3)) is 2: 1: This is the case of 1: 1.
In this way, it is possible to adaptively change the screen size according to the number of screens and form all the screens in the screen. This is an example in which a person shown on the screen (S (0)) is speaking, and when one person is speaking in this way, only that person's screen will depend on its size. The screen size can be adaptively enlarged / reduced.

In the screens S (0), S (1), S (2), S (3)) having an elliptical reference shape, the number of screens formed on the screen is increased or decreased, and the sound volume of one of the screens is increased. FIGS. 9A to 9C show examples in which the sound volume of other screens is halved.
9A to 9C, the number of screens is 2, 3, and 4 from the left, and the audio volume ratio (V (0): V (1)) is 2: 1. (V (0): V (1): V (2)) is 2: 1: 2, ratio (V (0): V (1): V (2): V (3)) is 2: 1: This is the case of 2: 2.
In this way, it is possible to adaptively change the screen size according to the number of screens and form all the screens in the screen. This is an example where a person other than the person shown on the screen (S (0)) is speaking, and in this way, even when multiple persons are speaking, the screen is adaptively adapted to the situation. It becomes possible to enlarge / reduce the size.

FIG. 10A shows an example in which the reference shape includes an ellipse (S (oval)), a circle (S (circle)), and a rectangle (S (rectangle)) (sound volume is equal). Shown in (D).
10A to 10D, from the left, a case where a rectangle and an ellipse, a circle and an ellipse, a circle and a rectangle, and a lower part are a circle, an ellipse, and a rectangle are shown.
Regardless of the shape of the reference shape, by setting the area to be equal, each screen adaptively adjusts the screen size, and if the audio volume is equal, each screen size is visually displayed equally It becomes possible to do.

Furthermore, in this embodiment, since the dead zone outside the screen is reduced, as shown in FIGS. 11 (A) to (C), an area separation line (bold line) is formed between the screens. The area based on the separation line is the display area of each screen. This makes it possible to divide and use the screen to the maximum while adapting to the increase / decrease in the number of screens and the increase / decrease in the sound volume of each screen.
In the example of FIGS. 11A to 11C, the audio volume ratio (V (0): V (1)) is 2: 1 and the ratio (V (0): V (1): V (2). )) Is 2: 1: 1 and the ratio (V (0): V (1): V (2): V (3)) is 2: 1: 1: 1.

  Next, the screen display control in the display unit 310 in which the size and the display form are controlled as described above will be described with reference to FIGS.

  In this embodiment, as described above, the image is divided by a circular (elliptical) window without a dead zone.

As shown in FIG. 12 and FIG. 13, on the screen to be displayed on the screen, the facial feature points are extracted from the image on the transmission side (encoding device side) or the reception side (decoding device side). The face area is calculated. An image is cut out and mapped to the screen so that the face area is included on the screen.
As shown in FIGS. 12 and 13, a facial feature point 111 is searched from the received image, and a contour is extracted to extract a face. Once the face area is extracted, the face area is tracked according to the motion vector, and cut out.

In the present embodiment, as shown in FIG. 14 associated with FIG. 2 and FIG. 3, the face area is detected on the encoding device 20 side (encoding side), and the detected face area is included and the face is detected. Select a circular screen to minimize the area other than. A rectangular area including a circular screen is cut out as a transmission image, encoded, and transmitted as a packet together with screen information and volume information of input sound such as a microphone.
In the encoding device 20, as shown in FIG. 15, the encoded portion is a part of the captured image. The shaded portion is a cut-off portion and is deleted from the encoding target. As a result, the capacity of image data to be transmitted is reduced.

Further, as shown in FIG. 16, the present invention can be applied to the case where the captured content is analyzed and the screen shape is changed according to the captured content.
In the example of FIG. 16, when it is determined that a person is captured, the screen shape is an ellipse, and in other cases, the screen shape is a rectangle.
In the example of FIG. 16, when the area of the face area is equal to or greater than a certain value, the received image is determined as a “person image”. When the area of the face area is equal to or smaller than a certain value, the received image is determined as a “non-person image”.
When it is determined that the image is a “person image”, a circular screen is used. When it is determined that the image is a “non-person image”, a quadrangular screen is used. Even for a “non-human image”, the display size is changed in accordance with the sound pressure from the transmission source.

As shown in FIG. 14, the decoding device 30 masks the received image decoded by the video decoding processing unit 305 from the received data based on the screen information extracted by the control information analysis unit 3061.
The display magnification calculator 3063 calculates the display magnification based on the size information, and the masked image is reduced or enlarged according to the magnification calculated by the reduction / enlargement processing unit 3064. On the other hand, the display position calculation unit 3065 calculates a display position according to the calculated display magnification, and displays a screen including an image reduced or enlarged at the calculated display position on the display unit 310.

Considering the load in processing when multipoint communication is performed individually and the terminal is configured as shown in FIGS.
Processing when the number of decoding devices 30 increases to the number N is as follows.

  Onnew = N × (masking process + reduction / enlargement process + mapping process + display magnification calculation + display position calculation)

  On the other hand, the processing when the decoding device side is configured to perform the transmission side (encoding device side) processing of the present embodiment on the decoding device side is as follows.

  Oold = N × (top and bottom correction process + face area detection + screen determination + cutout process + size calculation + reduction / enlargement process + display magnification calculation + display position calculation + mapping process)

  Looking at the difference in processing, the load shown in FIGS. 2 and 3 is reduced by the following amount.

Osub = Oold-Onew
= N × (top and bottom correction process + face area detection + screen determination + cutout process + size calculation−masking process)

  Of these, most of the processing load is “face area detection”.

  On the transmission side (encoding device side), the load is increased by the following amount.

  Enew = top and bottom correction processing + face area detection + screen determination + cutout processing + size calculation

  However, this load does not depend on the number of connected units.

As described above, according to the present embodiment, the encoding device 20 serving as the transmission source transmits the corresponding instruction information, screen information, and volume information when the transmission source has a screen at the designated position on the screen. The decoding device 30 has a function of displaying a multi-screen, a line connecting between the centers of the screen, a thickness of a reference shape, and a voice size. The screen display magnification is calculated based on the display magnification, and the screen movement and new generation are controlled based on the display magnification, thereby having the function of optimally forming a plurality of screens on the screen. The size can be adaptively changed according to the volume level and the number of screens.
In addition, there is an advantage that the movement of the screen becomes continuous, and even when the shapes are different, it can be arranged in an optimum size.
As a result, even when multiple terminals are connected, it is easy to check the other party in a call, and the image (screen) is controlled so that it does not overlap, so you can check the status of everyone who is talking at a glance can do. In addition, new participants can be easily accommodated.

It is a figure which shows the example of a display screen of terminals, such as a personal computer (PC) at the time of multipoint communication in a general video conference system. It is a figure which shows the structural example of the portable communication terminal which concerns on embodiment of this invention, Comprising: It is a block diagram which shows an encoding apparatus. It is a figure which shows the structural example of the portable communication terminal which concerns on embodiment of this invention, Comprising: It is a block diagram which shows a decoding apparatus. It is a figure for demonstrating the calculation process of a display magnification. It is a figure for demonstrating the wall process of a screen four directions. It is a figure which shows the example which changed the ratio of the magnitude | size (V (0), V (1)) of an audio | voice in the screen (S (0), S (1)) which makes an elliptical reference | standard shape. It is a figure which shows the example which increased / decreased the number of screens formed on a screen in the screen (S (0), S (1), S (2), S (3)) which makes an oval reference shape. In a screen (S (0), S (1), S (2), S (3)) with an elliptical reference shape, the number of screens formed on the screen is increased or decreased, and the sound of one of these screens is recorded. It is a figure which shows the example which doubled the magnitude | size of the audio | voice of the other screen. In the screens S (0), S (1), S (2), S (3)) having an elliptical reference shape, the number of screens formed on the screen is increased or decreased, and the sound volume of one of the screens is increased. It is a figure which shows the example which doubled the magnitude | size of the audio | voice of the other screen. It is a figure which shows the example when a reference | standard shape has mixed ellipse (S (oval)), circle | round | yen (S (circle)), and a rectangle (S (rectangle)) (the magnitude | size of an audio | voice is equal). It is a figure which shows the example which forms the separation line (bold line) of an area between each screen, and makes the area based on the said separation line into the display area of each screen. It is a figure for demonstrating screen display control, Comprising: It is explanatory drawing of the extraction process of a face area from a received image. It is a figure for demonstrating screen display control, Comprising: It is explanatory drawing of the clipping process after the extraction process of a face area from a received image. It is a figure which shows the operation | movement of this embodiment typically. It is a figure for demonstrating the operation | movement of this embodiment, Comprising: It is a figure which shows the transmission area example of the captured image in the transmission side. It is a figure for demonstrating screen display control, Comprising: It is explanatory drawing of the process according to the area of a face area.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Portable communication terminal, 20 ... Encoding apparatus, 201 ... Audio | voice input part, 202 ... Image input part, 203 ... Operation part, 204 ... Voice encoding process part, 205. ..Image encoding processing unit, 206... Top and bottom correction unit, 207... Face area detection unit, 208... Screen determination unit, 209. 211: Terminal control unit, 212: Control information generation unit, 213 ... Storage unit, 214 ... Transmission packet generation unit, 215 ... Network interface (I / F), 30 ... Decoding Device 301 network interface (I / F) 302 operation unit 303 received packet analysis unit 304 audio decoding processing unit 305 video decoding processing unit 306 ··table Image control unit, 307... Volume correction unit, 308 .. Audio output unit, 309... Image correction unit, 310... Display unit (image output unit), 311. ..Control information analysis unit, 3062 ... Masking processing unit, 3063 ... Display magnification calculation unit, 3064 ... Reduction / enlargement processing unit, 3065 ... Display position calculation unit, 3066 ... Mapping processing unit .

Claims (9)

A communication terminal that reproduces received image data and audio data,
Display means for displaying an image;
It is possible to form a plurality of display areas for displaying each of a plurality of images that should be displayed by extracting a specific area on the display means, and at least a line segment connecting the centers of the display areas of the images, a thickness of the reference shape, a sound A display magnification of the display area is calculated based on the size of the display area, and movement and new generation of the display area are controlled based on the display magnification to form a plurality of display areas on the display screen of the display means. And a communication terminal. The display area includes a center position coordinate (P (i)) indicating the display position of the display area, a reference shape (Unit (i)) indicating the shape of the display area, and a large amount of sound associated with the display area. (V (i)), and display magnification (R (i)) when displaying the display area on the screen,
The control means, as a display magnification (R (i)), a line segment (L (i, j)) connected to the center position coordinates (P (j)) of the surrounding screen, and the reference shape on the line segment Temporary display magnification (R (i, j, i)) calculated based on thickness (Lm (i, j), Lm (j, i)) and audio volume (V (i), V (j)) The communication terminal according to claim 1, wherein j)) is the smallest value. The control means sets the loudness (V (k) = 0) and thickness (Lm (k, i) = 0) at the point perpendicular to the screen boundary from the center of the display area, and displays it. The communication terminal according to claim 1 or 2, wherein a magnification (R (i, k)) is calculated. The communication terminal according to any one of claims 1 to 3, wherein the control means moves the display area to a position where the display magnification (R (i)) is maximized. The communication terminal according to any one of claims 1 to 4, wherein the control means generates a center of a new display area at a position where the display magnification (R (k)) is the largest. The communication terminal according to any one of claims 1 to 5, wherein the control means equalizes an area of the reference shape. The communication terminal according to any one of claims 1 to 6, wherein the control unit draws a separation line between screens formed in a reference shape, and uses the area separated by the separation line as a new display area. The information about the image to be displayed is included in the reception information from the transmission side device,
The communication terminal according to any one of claims 1 to 7, wherein the control unit calculates display magnification based on the reception information, and controls movement and new generation of a display area. A display method of a communication terminal that reproduces received image data and audio data,
Based on the line segment connecting the display area centers of a plurality of images to be extracted and displayed a specific area, the thickness of the reference shape, the volume of the sound, the display area display magnification is calculated,
Based on the display magnification, the display area is moved and newly generated, and a plurality of display areas are formed on the display screen.
A display method of a communication terminal that displays a plurality of display areas including an image to be displayed.

Images