JP7080614B2 - Image processing equipment, image processing system, image processing method, and program - Google Patents

Description

The present invention relates to a technique for extracting a specific region from a photographed image.

There is a technique called foreground background separation that extracts a region including a predetermined object (subject) in a captured image as a foreground region from the captured image. According to this technique, for example, an image of a moving person included in a captured image can be automatically obtained. Background subtraction methods include background subtraction, which extracts the foreground area based on the difference between the captured image and the background image stored in advance, and the foreground, which is based on the difference between multiple captured images taken continuously. There is an interframe subtraction method that extracts areas.

Patent Document 1 describes that the background subtraction method updates a background image according to a change in the brightness of the shooting environment to prevent erroneous detection of a moving object from the shot image.

Japanese Unexamined Patent Publication No. 2000-324477

However, in the conventional technique, a predetermined subject area to be extracted and another area may be extracted without distinction based on the difference between a plurality of images taken at different timings. .. For example, when a moving object to be extracted and a display whose display contents change with time are included in the captured image, the area of the moving object and the area of the display are similarly extracted.

The present invention has been made in view of the above problems, and a region of a predetermined subject to be extracted and a region different from the region of a predetermined subject to be extracted are distinguished based on the difference between a plurality of images based on shooting at a plurality of different timings. The purpose is to prevent it from being extracted without any problem.

In order to solve the above problems, the image processing apparatus according to the present invention has, for example, the following configuration. That is, it is an image processing device that extracts a subject area corresponding to a predetermined subject from an image based on the image taken by the photographing device, and is based on the photographing device at a plurality of timings within a first period designated by the first predetermined operation. Based on the difference between a plurality of images based on shooting, the area inside the image to be extracted based on shooting by the shooting device within the second period specified by the second predetermined operation, and the target of extraction of the subject area. The extraction target image is based on the difference between the specific means for specifying a region not to be extracted and another image based on the imaging by the imaging device at a timing different from the imaging timing of the extraction target image. It is an extraction means for extracting the subject area inside, and has an extraction means for extracting the subject area composed of pixels not included in the area specified by the specific means.

According to the present invention, it is possible to suppress that a predetermined subject to be extracted and a region different from the predetermined subject are extracted without distinction based on the difference between a plurality of images taken at a plurality of different timings. It will be possible.

It is a block diagram which shows the structure of an image processing system 100. It is a figure for demonstrating the change of the photographed image in an embodiment. It is a figure for demonstrating the background change area in an embodiment. It is a flowchart for demonstrating the process of detecting the background change area by an image processing apparatus 100. It is a figure for demonstrating the background image and the foreground image in an embodiment. It is a flowchart for demonstrating the process of the foreground background separation by an image processing apparatus 100.

[System configuration]
FIG. 1A is a diagram for explaining a schematic configuration of an image processing system 10 according to an embodiment. The image processing system 10 includes an image processing device 100, a photographing device 110, and an image processing server 120.

The photographing device 110 generates a photographed image by performing photography, and inputs the photographed image to the image processing device 100. The photographing device 110 is, for example, a digital video camera provided with an image signal interface such as a serial digital interface (SDI) for inputting a captured image. The captured image in the present embodiment includes an image that has undergone image processing such as filter processing and resolution conversion after shooting.

The image processing device 100 performs image processing on the captured image input from the photographing device 110, extracts the foreground region from the captured image, and separates the captured image into a foreground region and a background region. In the present embodiment, this process is referred to as foreground background separation. In the present embodiment, the foreground area is a subject area corresponding to a predetermined object (subject) in the captured image, and the background area is an area not corresponding to the predetermined subject. For example, the image processing device 100 acquires a photographed image taken by the photographing apparatus 110 in a stadium where a soccer game is played, and the acquired photographed image is used as a foreground region including a predetermined subject such as a player, a referee, or a ball. And the background area including the field surface and the audience seats. Then, the image processing apparatus 100 outputs the foreground image based on the foreground region and the background image based on the background region to the image processing server 120.

The detailed configuration of the image processing device 100 will be described later. The shooting target by the shooting device 110 is not limited to soccer, but may be other sports such as rugby or sumo, or may be a live performance on a stage. Further, the predetermined subject extracted as the foreground region by the image processing device 100 is not limited to the player or the ball.

The image processing server 120 performs image processing based on the image input from the image processing device 100. For example, the image processing server 120 acquires a foreground image and a background image from the image processing device 100 via a network cable, generates an image for display, and displays the image on a display unit (not shown).

In the present embodiment, the image processing system 10 has a plurality of photographing devices 110 and a plurality of image processing devices 100 as shown in FIG. 1 (a). The plurality of photographing devices 110 are installed in, for example, a stadium to be photographed, and photograph from different directions. The plurality of image processing devices 100 acquire captured images from the corresponding photographing devices 110, perform foreground background separation, and output the foreground image and the background image to the image processing server 120. That is, the image processing server 120 acquires a plurality of foreground images and a plurality of background images obtained by extraction processing for each of the captured images by the plurality of photographing devices. Then, the image processing server 120 generates a virtual viewpoint image including a predetermined subject extracted as a foreground region. In FIG. 1A, the image processing system 10 includes two photographing devices 110, but the number of photographing devices 110 is not limited to this, and may be three or more.

The virtual viewpoint image in the present embodiment represents an image obtained when a subject is photographed from a virtual viewpoint (virtual viewpoint). In other words, a virtual viewpoint image is an image that represents the field of view at a specified viewpoint. The virtual viewpoint may be designated, for example, by the user of the image processing server 120, or may be automatically designated based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to the viewpoint arbitrarily designated by the user. Further, an image corresponding to a viewpoint designated by the user from a plurality of candidates and an image corresponding to the viewpoint automatically designated by the device are also included in the virtual viewpoint image. In the present embodiment, unless otherwise specified, the word "image" will be described as including the concepts of both moving images and still images. That is, the image processing system 10 of the present embodiment can process both still images and moving images.

In order to generate the virtual viewpoint image, the image processing server 120 acquires the viewpoint information according to the designation of the virtual viewpoint. Further, the image processing server 120 obtains a three-dimensional model of a predetermined subject extracted as a foreground region based on a plurality of foreground images acquired from a plurality of image processing devices 100 corresponding to a plurality of photographing devices 110 having different shooting directions. Generate. A known method is used for 3D model generation, for example, a method using Visual Hull. Then, the image processing server 120 performs rendering based on the acquired viewpoint information, the three-dimensional model, and the background image, and generates a virtual viewpoint image including a predetermined subject.

The method for generating the virtual viewpoint image by the image processing server 120 is not limited to the method using the three-dimensional model, and may be another method. For example, the image processing server 120 may generate a virtual viewpoint image by projecting and transforming the acquired foreground image and background image based on the viewpoint information and synthesizing the converted foreground image and background image. Further, the process performed by the image processing server 120 is not limited to the generation of an image such as a virtual viewpoint image, but may be a process of displaying the acquired foreground image itself, or supports a foreground image, a background image, a three-dimensional model, or the like. It may be a process of attaching and outputting to an external database.

As described above with reference to FIG. 1A, in the image processing system 10 of the present embodiment, the foreground and background separation for the images captured by the plurality of imaging devices 110 is dispersed among the plurality of image processing devices 100. conduct. As a result, the load on the image processing server 120 can be reduced and the delay related to the processing of the image processing system 10 as a whole can be reduced as compared with the case where the foreground background separation is performed collectively on the image processing server 120. The configuration of the image processing system 10 is not limited to that described above. For example, a single image processing device 100 may acquire captured images from a plurality of imaging devices 110 and perform foreground background separation for each captured image. Further, the image processing device 100 and the image processing server 120 may be integrally configured, or the components of the image processing device 100, which will be described later, may be divided into a plurality of devices.

[Device configuration]
FIG. 1B is a diagram for explaining a hardware configuration of the image processing apparatus 100 according to the present embodiment. The configuration of the image processing server 120 is the same as that of the image processing device 100. The image processing device 100 includes a CPU 111, a RAM 112, a ROM 113, an input unit 114, an external interface 115, and an output unit 116.

The CPU 111 controls the entire image processing apparatus 100 by using computer programs and data stored in the RAM 112 and the ROM 113. The image processing device 100 may have one or more dedicated hardware or GPU (Graphics Processing Unit) different from the CPU 111, and the GPU or dedicated hardware may perform at least a part of the processing by the CPU 111. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The RAM 112 temporarily stores computer programs and data read from the ROM 113, data supplied from the outside via the external interface 115, and the like. The ROM 113 holds computer programs and data that do not need to be changed.

The input unit 114 is composed of, for example, an operation button, a jog dial, a touch panel, a keyboard, a mouse, and the like, and receives an operation by a user and inputs various instructions to the CPU 111. The external interface 115 communicates with an external device such as a photographing device 110 or an image processing server 120. Communication with an external device may be performed by wire using a LAN (Local Area Network) cable, SDI cable, or the like, or may be performed wirelessly via an antenna. The output unit 116 is composed of, for example, a display unit such as a display and an audio output unit such as a speaker, and displays a GUI (Graphical User Interface) for a user to operate the image processing device 100 and outputs a guide sound. do.

Next, the details of the functional configuration of the image processing apparatus 100 shown in FIG. 1A will be described. The image processing device 100 has a foreground background separation unit 101 (hereinafter, separation unit 101), a change area detection unit 102 (hereinafter, detection unit 102), and a communication unit 103. Each of these functional units included in the image processing device 100 is realized by the CPU 111 expanding the program stored in the ROM 113 into the RAM 112 and executing the program. It should be noted that at least a part of the functional unit of the image processing device 100 shown in FIG. 1A may be realized by one or a plurality of dedicated hardware or GPU different from the CPU 111.

The separation unit 101 separates the foreground and background from the captured image input from the photographing device 110, and outputs the foreground image and the background image to the communication unit 103. In the present embodiment, the separation unit 101 extracts a foreground region in the captured image by using the background subtraction method. In the background subtraction method, a captured image including a subject to be extracted is compared with a background image stored in advance and not including the subject, and a region in which the difference in pixel values is larger than the threshold value is extracted. For example, by comparing the captured image taken during the match in which the player to be extracted is included in the shooting range with the background image taken before the match in which the player does not exist in the shooting range, the player in the captured image is included. The area corresponding to is extracted. The method of separating the foreground and background by the separation unit 101 is not limited to this, and for example, the inter-frame difference method may be used. In the inter-frame difference method, a region based on the difference between a plurality of captured images continuously captured by the same imaging device is extracted.

Here, when the separation unit 101 uses the conventional background subtraction method as it is, it may not be possible to extract only the area of the subject to be extracted. For example, consider a case where it is desired to extract an area in which a player is reflected from a photographed image of a soccer game in a stadium. FIG. 5A shows an example of a captured image 5000 to be extracted. In addition to the athletes 5001 and the field 2001, the captured image 5000 shows the display 2002, the display 2003, and the display 2004 for displaying advertisements installed on the side of the field. The displays 2002-2004 each display an image that changes with the passage of time.

In this case, when the background subtraction method is executed on the captured image 5000 with the captured image captured in advance as shown in FIG. 5 (b) as the background image 5100, as shown in FIG. 5 (c). A difference region image 5200 is obtained. The difference region image 5200 in the present embodiment is specifically an image composed of pixels in the captured image 5000 whose pixel value difference from the corresponding pixel in the background image 5100 is larger than the threshold value. .. However, the difference area image 5200 may be an image in a region where the difference between the captured image 5000 and the background image 5100 is equal to or greater than the threshold value, and is not limited to the one specified by calculating the difference in pixel values for each pixel. .. For example, the difference between the captured image 5000 and the background image 5100 may be calculated for each block composed of a plurality of pixels. The difference between the corresponding blocks in a plurality of images may be calculated by using the average value of the pixel values in the block, or may be calculated by using the mode value of the pixel values in the block or the like.

As shown in FIG. 5C, when the conventional background subtraction method is executed, the area corresponding to the display surface of the display 2002-2004 is similarly extracted together with the area corresponding to the player 5001. If these areas are similarly extracted, the image quality of the virtual viewpoint image generated by the image processing server 120 based on the extraction result may be deteriorated. For example, when the image processing device 100 generates a difference region image from each of a plurality of captured images obtained by photographing the field 2001 and its surroundings from a plurality of directions by a plurality of photographing devices, the player 5001 is included in any of the difference region images. Is done. On the other hand, since the display surface of the display 2002-2004 is not included in the captured image captured from the back side direction of the display 2002-2004, it is not included in the difference region image generated from the captured image. It is difficult to accurately associate the pixels of the difference region image including both the athlete 5001 and the display surface with the pixels of the difference region image including only the athlete 5001. Therefore, when the image processing server 120 tries to generate a three-dimensional model of the player 5001 based on these difference region images, the accuracy of the model deteriorates. As a result, the image quality of the virtual viewpoint image also deteriorates.

Therefore, the image processing apparatus 100 in the present embodiment acquires a foreground image 5300 as shown in FIG. 5D, which includes an area corresponding to the player 5001 and does not include an area corresponding to the display surface of the display 2002-2004. Therefore, a detection unit 102 is provided. The detection unit 102 detects the background change area in the captured image. The background change area is an area that is not the target of extraction as the foreground area. More specifically, the background change region is a region that should be identified as a background region, that is, a region that should not be extracted as a foreground region, but that changes with the passage of time. In the case of the example shown in FIG. 5 above, the area corresponding to the display surface of the display 2002-2004 is the background change area. However, the subject is not limited to this, and the subject corresponding to the background change area may be a subject different from the predetermined subject to be extracted and may be a subject that changes with the passage of time. The method of detecting the background change area will be described later.

When the detection unit 102 detects the background change region, the detection unit 102 provides the separation unit 101 with information indicating the detected region. Then, the separation unit 101 can extract the region of the subject to be extracted as the foreground region by performing the foreground background separation on the captured image acquired from the imaging device 110 using the information acquired from the detection unit 102. The details of the foreground background separation by the separation unit 101 will be described later.

The communication unit 103 transmits the foreground image and the background image input from the separation unit 101 to the image processing server 120. The communication unit 103 is composed of, for example, a LAN card provided with a high-speed serial interface such as PCI Express.

[Detection of background change area]
Next, the detection of the background change area will be described. 2 (a)-(c) show an example of the photographed image photographed by the photographing apparatus 110. 2A is a captured image 2000 captured by the photographing device 110 at time T, FIG. 2B is a captured image 2100 captured by the photographing device 110 at time T + 1, and FIG. 2C is a photographing device. 110 is a captured image 2200 taken at time T + 2.

The subject to be photographed is a soccer game, and in the photographed image 2000, the field 2001 and the display 2002-2004 for displaying an advertisement are photographed. A line 2005 indicating the goal area is drawn in the field 2001. The captured image 2000-2200 is a situation in which there is no person such as a player in the shooting range, such as a scene where a match is being prepared at a stadium, and the display surface of the display 2002-2004 is included in the shooting range. It is an image taken in. At the time of this shooting, the advertisement display is being rehearsed, and the image displayed on the displays 2002-2004 changes.

The display 2002 for displaying the advertisement displays the advertisement image 2012 at the time T, while displaying the advertisement image 2112 at the time T + 1. For example, the advertisement image 2112 is an image in the middle of changing to the next advertisement image 2212 with the image effect that the advertisement image 2012 scrolls vertically. Similarly, the advertisement images 2013 and 2014 are changed to the advertisement images 2113 and 2114, respectively. The advertisement image 2113 is an image in the middle of changing to the next advertisement image 2213 with the image effect that the advertisement image 2013 scrolls horizontally.

FIG. 3A shows an example of the difference between the pixel values of the captured image 2000 and the captured image 2200. For example, the pixel values (R, G, B) of the pixels corresponding to the display surface of the display 2002 in the captured image 2000 at time T are (220, 10, 10). Further, the pixel values (R, G, B) of the pixels corresponding to the display surface of the display 2003 and the pixel values (R, G, B) of the pixels corresponding to the display surface of the display 2004 are (10, 230, 10), respectively. ) And (10, 10, 240).

On the other hand, the pixel values of the pixels corresponding to the display surface of the display 2002-2004 in the captured image 2200 at time T + 2 are (10, 230, 10), (10, 10, 240), and (220, 10, 10), respectively. be. In FIG. 3, for the sake of simplicity, the images displayed on the displays 2002-2004 are assumed to have the same pixel value on the entire display surface at the same time. That is, the pixel values of the pixels corresponding to the display surface of the display 2002 in a single captured image are uniquely represented. The same applies to the display 2003 and the display 2004. However, the present invention is not limited to this, and may be a display image having different pixel values depending on the position in the display surface, for example, as shown in FIG. Further, the color space representing the captured image is not limited to RGB, and may be another color space.

Next, the operation of the detection unit 102 to detect the background change region will be described with reference to FIG. The process shown in FIG. 4 is started at the timing when the captured image is acquired from the photographing device 110 in the mode in which the image processing device 100 detects the background change region (S4010). However, the start timing of the process shown in FIG. 4 is not limited to this. The mode of the image processing device 100 is set according to, for example, an operation by the user. Specifically, the image processing apparatus 100 is set to the mode for detecting the background change area by designating the shooting period of the image for detecting the background change area according to a predetermined operation by the user. The shooting period specified here is a period during which a predetermined subject such as a player is not included in the shooting range of the shooting device 110, for example, during a rehearsal before the start of a match at a stadium. The user may perform operations at the start and end of the shooting period, respectively.

Since the shooting period of the image for detecting the background change area can be specified by the user's operation, it is possible to set an arbitrary period suitable for detecting the background change area. For example, by using a captured image that does not include a predetermined subject to be extracted as the foreground for detecting a background change area, a captured image that includes both a subject to be extracted and a subject that is not to be extracted is used as compared to the case where a captured image is used. , The background change area can be specified accurately.

In the following description, it is assumed that the captured image input from the photographing device 110 to the image processing device 100 is each frame of the moving image, but the input captured image is a plurality of still images taken at a plurality of time points. May be good. The process shown in FIG. 4 may be performed in real time in parallel with the image taken by the photographing apparatus 110, or may be performed after the image taken based on the accumulated photographed image. In any case, FIG. 4 is based on a photographed image taken in a situation where a predetermined subject to be extracted is not included in the photographing range of the photographing apparatus 110, such as before the start of a game in which there is no player 5001 in the field 2001. The process is executed.

The process shown in FIG. 4 is realized by the CPU 111 expanding the program stored in the ROM 113 into the RAM 112 and executing the program. It should be noted that at least a part of the processing shown in FIG. 4 may be realized by one or a plurality of dedicated hardware or GPU different from the CPU 111.

In S4020, the detection unit 102 determines the target frame from the frame of the moving image acquired from the photographing device 110 in the mode of detecting the background change region, and determines the difference in the pixel value between the target frame and the frame at the time point before the target frame. calculate. For example, the detection unit 102 compares the captured image 2000 shown in FIG. 2 (a) captured at time T with the captured image 2200 shown in FIG. 2 (c) captured at time T + 2, and the pixels of the corresponding pixels. Calculate the difference between the values. The target frame in this case is the captured image 2200. When the pixel value of a certain pixel in the captured image 2000 is (R0, G0, B0) and the pixel value of the corresponding pixel in the captured image 2200 is (R1, G1, B1), the pixel value is calculated by the following equation (1). The difference value δd (absolute value of the difference) is obtained.
δd = | R0-R1 | + | G0-G1 | + | B0-B1 | ... (1)

FIG. 3A shows the difference value of each pixel calculated from the captured image 2000 and the captured image 2200. The pixel value of the pixel corresponding to the display surface of the display 2002 is (220,10,10) at time T and (10,230,10) at time T + 2. Therefore, the difference value δd of the pixel values of the pixels corresponding to the display surface of the display 2002 is 430 by the following calculation.
δd = | 220-10 | + | 10-230 | + | 10-10 | = 430 ... (2)

The detection unit 102 calculates the above difference value for all the pixels of the target frame. Among the calculated difference values, the difference values of the pixels corresponding to the display 2003, the display 2004, the field 2001, and the line 2005 are, for example, the values shown in FIG. 3A. Specifically, the difference value of the pixel corresponding to the display surface of the display 2003 and the difference value of the pixel corresponding to the display 2004 are 450 and 440, respectively, by the calculation shown in the equation (1). When the image displayed by the displays 2002-2004 has different pixel values depending on the position in the display surface, a different difference value can be calculated for each position in the display surface.

On the other hand, for the pixels corresponding to the field 2001 and the pixels corresponding to the line 2005, the pixel values are almost the same between the captured image 2000 at time T and the captured image 2200 at time T + 2, unlike the dynamically changing display surface. It does not change. Specifically, as shown in FIG. 3A, the pixel value of the pixel corresponding to the field 2001 at time T is (180, 230, 30), and the pixel value at time T + 2 is (178, 228, 28). It becomes. Therefore, the difference value δd of the pixel values of the pixels corresponding to the field 2001 is 6 by the following calculation.
δd = | 180-178 | + | 230-228 | + | 30-28 | = 6 ... (3)

For the pixels corresponding to the line 2005, the difference value δd is 3 by the same calculation. When the difference value is calculated for all the pixels of the target frame, the process transitions to S4030. In this embodiment, the difference value is calculated for all the pixels of the target frame, but the present invention is not limited to this. For example, when the range that is a candidate for the background change area is preset or specified by the user, the detection unit 102 detects the set range or the pixels included in the specified range. Only the difference value may be calculated.

In S4030, the detection unit 102 integrates the difference values δd calculated for the plurality of target frames. Here, since the detection unit 102 performs the processing of S4030 for the first time in the flow of FIG. 4, the pixel difference value δd calculated in the above-mentioned S4010 is stored as an integrated value.

In S4040, the detection unit 102 determines whether or not the integration of the difference values δd for the predetermined number of target frames is completed. In the present embodiment, the number of frames targeted by the detection unit 102 for integration is 100. The number of frames to be integrated is set based on, for example, an operation by the user. However, the present invention is not limited to this, and the number of frames to be integrated may be automatically set according to, for example, the amount of change of the captured image with the passage of time. When it is determined that the integration of the difference value δd for the predetermined number of target frames has not been completed, the detection unit 102 changes the target frame to a frame after the frame in which the difference value is calculated, and returns to S4020. ..

For example, the detection unit 102 calculates the difference in pixel value between the captured image 2200 captured at time T + 2 and the captured image captured at time T + 4. Here, for the sake of simplicity, it is assumed that the difference in pixel values between time T + 2 and time T + 4 is the same as the difference in pixel values between time T and time T + 2 described above. That is, the difference values δd of the pixel values between the time T + 2 and the time T + 4 of the pixels corresponding to the display surfaces of the display 2002, the display 2003, and the display 2004 are 430, 450, and 440, respectively. Further, the difference values δd of the pixel values of the pixels corresponding to the field 2001 and the line 2005 are 6 and 3, respectively. Then, the detection unit 102 adds the calculated difference value to the integrated value stored in S4030.

The detection unit 102 repeats the calculation of the difference value in S4020 and the addition to the integrated value in S4030 for 100 frames, which is a predetermined number of frames. As a result of integrating the difference values for 100 frames, the integrated values of the difference values δd of the pixels corresponding to the display surfaces of the display 2002, the display 2003, and the display 2004 are 43000, 45000, and 44000, respectively. Further, the integrated values of the difference values δd of the pixels corresponding to the field 2001 and the line 2005 are 600 and 300, respectively.

An example of the integrated value of the difference value δd calculated by the detection unit 102 will be described with reference to FIG. 3 (b). FIG. 3B shows the integrated values for the pixels on the line segment 2105 in the captured image 2100 of FIG. 2B. The horizontal axis represents the horizontal pixel position of the captured image, and the vertical axis represents the integrated value.

The horizontal coordinate values of the pixels corresponding to the display surface of the display 2002 are 300 to 750, and the integrated value of the difference value δd of these pixels is 43000. On the other hand, the integrated value of the difference value δd of the pixels (pixels having horizontal coordinate values of 0 to 299, 751 to 899, and 1351 to 1499) that do not correspond to the display surface of the display 2002-2004 is 600.

When it is determined in S4040 that the integration of the difference values δd for a predetermined number of target frames is completed, the process transitions to S4050. In S4050, the detection unit 102 evaluates each pixel to see if the integrated value is equal to or greater than the threshold value. For example, when the integrated value as shown in FIG. 3B is obtained, the evaluation is performed with the value of the threshold value 3101 as 30,000. The threshold value 3101 may be set according to the operation of the user, or may be automatically set based on the average of the integrated values and the like.

As described above, since the integrated value of the difference value δd of the pixels corresponding to the display surface of the display 2002 is 43000, it is 30,000 or more, which is the value of the threshold value 3101. Similarly, the integrated value of the pixels corresponding to the display surfaces of the display 2003 and the display 2004 is 30,000 or more. Then, the detection unit 102 detects the pixels whose integrated value is the threshold value 3101 or more as the background change region. As a result, the background change area including the area corresponding to the display surface of the display 2002-2004 is detected. On the other hand, the area other than the display 2002-2004, that is, the area corresponding to the field 2001 and the area corresponding to the line 2005 are not included in the background change area because the integrated value is less than the threshold value 3101.

When the detection process in S4050 is completed, the process shown in FIG. 4 is completed in S4100. As described above, the detection unit 102 is a plurality of detection units 102 at a plurality of different timings within a period designated by a predetermined operation by the user (for example, within a period during which a predetermined subject to be extracted is not included in the shooting range of the shooting device 110). Take a picture of. Then, the difference between the pixel values of the corresponding pixels in the plurality of captured images is calculated. Then, the detection unit 102 is a pixel position in the captured image captured by the imaging device 110, and the pixel position of the pixel whose calculated difference is equal to or larger than the threshold value is not the pixel position of the background change region to be extracted in the foreground region. Specify as a position.

In the present embodiment, the case where the image processing device 100 specifies the pixel position of the background change region based on the difference between a plurality of images will be mainly described, but the specifying method is not limited to this. For example, the image processing device 100 may specify the pixel position of the background change region according to the operation by the user.

FIG. 3C shows the detected background change region, and the region corresponding to the display surface of the display 2002-2004 is detected as the background change region 2301-2303. In the present embodiment, the detection unit 102 calculates the integrated value of the difference value δd as the difference between the pixel values of the corresponding pixels in a plurality of captured images having different imaging timings, and detects the background change region. Specifically, the detection unit 102 does not target the pixel position in which the amount of change in the pixel value during the shooting period of a plurality of time-continuously shot images by the same shooting device is equal to or greater than the threshold value to be extracted in the foreground region. It is specified as the pixel position. By using such a method, the background change region can be detected in various cases such as when the display image of the display 2002-2004 gradually changes or when the display image is periodically switched.

However, the present invention is not limited to this, and the detection unit 102 may detect the background change region by the background subtraction method. For example, a captured image in a situation where an image is not displayed on the displays 2002-2004 is used as a background image. Then, the detection unit 102 defines a region composed of pixels of the captured image in the situation where the image is displayed on the display 2002-2004, in which the difference in pixel value from the corresponding pixel of the background image is equal to or larger than the threshold value. , May be detected as a background change area. Even with such a method, a background change region similar to the method described with reference to FIG. 4 can be detected.

In addition, various other methods can be used as a method for detecting the area corresponding to the display 2002-2004 as the background change area. For example, the image processing device 100 may acquire information regarding at least one of the positions and shapes of the displays 2002-2004, and specify the background change region based on the acquired information. Specific examples of the information regarding at least one of the positions and shapes of the displays 2002-2004 include an image representing the display, an image displayed on the display, a three-dimensional model of the display, and information according to the operation by the user. Be done.

For example, the image processing device 100 may acquire an image representing the display 2002-2004 and detect the background change region by collating the acquired image with the captured image. Further, the image processing device 100 may detect the background change region by detecting the marker displayed on the display 2002-2004 from the captured image. Further, the image processing device 100 may detect the background change region by specifying the position of the display 2002-2004 with reference to design information such as a three-dimensional model of the stadium including the display 2002-2004. Further, the image processing device 100 displays an image for allowing the user to specify the position and shape of the display 2002-2004, and specifies the position and shape of the display 2002-2004 based on the designated operation by the user to change the background. Regions may be detected. Further, the image processing apparatus 100 may detect a background change region based on the luminance information of the pixels of the captured image. In addition, when specifying a region corresponding to a subject different from the display 2002-2004 as a background change region, various methods similar to the above can be used.

[Foreground background separation]
Next, the foreground background separation by the separation unit 101 will be described. FIG. 5A shows an example of a photographed image 5000 photographed by the photographing apparatus 110 and input to the separation unit 101. The captured image 5000 is an image captured in a scene where a game is being played at a stadium, unlike the captured image captured for detecting the background change area information described above. Therefore, the captured image 5000 is an image captured in a situation where the player 5001 is on the field 2001 within the photographing range and the display surface of the display 2002-2004 is included in the photographing range. The images displayed on the displays 2002-2004 are changing as in the rehearsal. The separation unit 101 extracts the region of the player 5001 included in the captured image 5000 by foreground background separation.

The operation of the foreground background separation by the separation unit 101 will be described with reference to FIG. The process shown in FIG. 6 is started at the timing when the captured image is acquired from the photographing device 110 in the mode in which the image processing device 100 performs the foreground background separation (S6010). However, the start timing of the process shown in FIG. 6 is not limited to this. The mode of the image processing device 100 is set according to, for example, an operation by the user. Specifically, the image processing apparatus 100 is set to the mode for performing the foreground background separation by designating the shooting period of the image to be the foreground background separation for detecting the subject area according to a predetermined operation by the user. Will be done. The shooting period designated here is a period in which a predetermined subject such as a player is included in the shooting range of the shooting device 110, for example, during a game in a stadium. The shooting period for separating the foreground and background is specified by an operation different from the operation for specifying the shooting period for detecting the background change area described in FIG. 4, but is not limited to this. The mode may be changed depending on how many times the operation is performed.

The process shown in FIG. 6 may be performed in real time in parallel with the image taken by the photographing apparatus 110, or may be performed after the image taken based on the accumulated photographed image. In any case, the processing of FIG. 6 is performed based on the captured image taken in a situation where a predetermined subject to be extracted is included in the shooting range of the shooting device 110, such as during a game in which the player 5001 is in the field 2001. Will be executed.

The process shown in FIG. 6 is realized by the CPU 111 expanding the program stored in the ROM 113 into the RAM 112 and executing the program. It should be noted that at least a part of the processing shown in FIG. 6 may be realized by one or a plurality of dedicated hardware or GPU different from the CPU 111.

In S6020, the separation unit 101 determines the target frame from the frame of the moving image acquired from the photographing device 110 in the mode of performing the foreground background separation, and calculates the difference between the captured image 5000 and the background image 5100 which are the target frames. The target frame is an image (extraction target image) to be extracted in the foreground region, and when S6020 is executed for the first time in the flow of FIG. 6, for example, it is determined to be the first frame of the acquired moving image. The background image 5100 is an image based on shooting by the shooting device 110 at a timing different from that of the shot image 5000 as the extraction target image, is stored in advance, and does not include a predetermined subject to be extracted.

When the process of S6020 is executed for the first time, for example, a captured image as shown in FIG. 5 (b) captured during the pre-match rehearsal is used as the background image 5100. That is, in this case, the image based on the shooting by the shooting device 110 during the shooting period for detecting the background change region described with reference to FIG. 4 becomes the background image. Then, by calculating the difference between the captured image 5000 and the background image 5100 using the background subtraction method, a difference region image 5200 as shown in FIG. 5C can be obtained. Specifically, the difference region image 5200 is an image composed of pixels in the captured image 5000 whose pixel value difference (absolute value of difference) from the corresponding pixel in the background image 5100 is equal to or larger than the threshold value. Is. The difference area image 5200 includes the advertisement image 2212-2214 displayed on the display surface of the display 2002-2004 in addition to the player 5001 to be extracted as the foreground area.

In S6030, the separation unit 101 acquires information indicating the background change region detected in the process described with reference to FIG. 4 from the detection unit 102. As described above, the detected background change region becomes, for example, the background change region 2301-2303 shown in FIG. 3 (c), which corresponds to the display surface of the display 2002-2004.

In S6040, the separation unit 101 generates the foreground image 5300 as shown in FIG. 5D by excluding the region overlapping the background change region 2301-2303 from the difference region image 5200. That is, the separation unit 101 extracts a region of the pixels included in the difference region image 5200, which is composed of pixels different from the pixel position specified as the background change region, as the foreground region. The foreground image 5300 does not include the advertising image 2212-2214 displayed on the display surface of the display 2002-2004, but includes only the player 5001.

In S6050, the separation unit 101 overwrites the pixel values of the pixels included in the background change area in the difference area image 5200 with the stored background image 5100. That is, the separation unit 101 updates the background image 5100 based on the pixels in the difference region image 5200 that are not included in the region extracted as the foreground region. By updating the background image 5100, a new background image 5400 as shown in FIG. 5 (e) can be obtained. The background image 5400 obtained here is an image based on shooting by the shooting device 110 within the shooting period for separating the foreground and background. Then, when the processing of S6020 is performed again in the flow of FIG. 6, the difference between the new target frame and the updated background image 5400 is calculated. The inter-frame difference method may be used to calculate the difference at this time.

When the area detected as the background change area is fixed, such as the display surface of the installed display 2002-2004, the separation unit 101 does not have to update the background image 5100. If the image data of the image displayed on the displays 2002-2004 can be acquired, the background image 5100 may be updated using the image data instead of using the difference region image.

In S6060, the separation unit 101 separates the image of the foreground region (foreground image 5300) extracted in S6040 and the background image 5400 generated in S6050 and outputs the image to the image processing server 120. By acquiring the foreground image 5300 and the background image 5400 as separated and identifiable images, the image processing server 120 generates a three-dimensional model of a predetermined subject and a three-dimensional model of the background, respectively, to generate a virtual viewpoint image. Can be generated.

When the processing of S6060 is completed, the process transitions to S6070, and the separation unit 101 determines whether the processing of S6020-S6060 is completed for all the frames of the moving image acquired from the photographing device 110 in the mode of performing the foreground background separation. When it is determined that the processing for all the frames is not completed, the separation unit 101 changes the target frame to the next frame and returns to S6020. On the other hand, when it is determined that the processing is completed for all the frames, the processing of FIG. 6 ends in S6500.

In the above description using FIG. 6, the separation unit 101 is composed of pixels in the captured image 5000 whose pixel value difference from the corresponding pixel in the background image 5100 is equal to or greater than the threshold value. The foreground area is extracted by removing the background change area from the area. However, not limited to this, the separation unit 101 includes a partial image that is a partial image of the captured image 5000 and is composed of pixels that are not included in the background change region, and a region corresponding to the partial image in the background image 5100. The foreground area may be extracted by determining the difference. According to this method, the number of pixels to be calculated for the difference between the pixel values is reduced, so that the processing load of the image processing apparatus 100 can be reduced. In either method, a foreground region composed of pixels not included in the background change region is extracted from the captured image 5000 based on the difference between the captured image 5000 and the background image 5100.

Further, in the present embodiment, there is a case where the background change area is specified based on the difference between a plurality of captured images captured by the imaging device 110 before the captured image 5000 to be the target of the process of extracting the foreground region. I explained mainly. However, the present invention is not limited to this, and for example, when the foreground area is extracted from the captured images 5000 captured during the match after the match, the background is based on the difference between the plurality of captured images captured after the match. The area of change may be identified.

As described above, the image processing device 100 according to the present embodiment extracts a subject area (foreground area) corresponding to a predetermined subject from an image based on the image taken by the image pickup device 110. Specifically, the image processing device 100 determines the subject area inside the extraction target image based on the difference between a plurality of images based on shooting at a plurality of timings within the first period designated by the first predetermined operation. Specify the area (background change area) that is not the target of extraction. Here, the extraction target image is an image based on photography by the photographing apparatus 110 within the second period designated by the second predetermined operation different from the first predetermined operation described above. Then, the image processing device 100 determines the subject area inside the extraction target image based on the difference between the extraction target image and another image based on the shooting by the shooting device 110 at a timing different from the shooting timing of the extraction target image. Extract. The subject area extracted in this way is composed of pixels not included in the area specified as the area not to be extracted.

According to such a configuration, it is possible to extract a specific region among the regions that are the differences between a plurality of images. For example, when a predetermined subject moving in the captured image and a display displaying a moving image are included, the image processing device 100 can extract only the region of the predetermined subject. Therefore, the image processing server 120 that acquires the extraction result from the image processing device 100 can accurately specify the shape of a predetermined subject, and can generate a high-quality virtual viewpoint image including the subject.

In the present embodiment, the case where the region corresponding to the display surface of the display 2002-2004 in the captured image is detected as the background change region and the region corresponding to the player 5001 is extracted as the foreground region has been mainly described. .. However, the background change area and the foreground area are not limited to these. For example, a subject such as a ball may be extracted as a foreground area. Further, as the background change area, a display surface on which an image is projected by a projector, an audience seat where an audience moves, or the like may be detected. Specifically, by detecting the spectator seat as a background change area from the photographed image taken while the spectator is moving before the start of the game, the spectator seat is used as the foreground area from the photographed image taken during the game. It may not be extracted.

Further, in the present embodiment, a case where the image processing device 100 generates and outputs a foreground image and a background image for each frame of the moving image has been described. However, the present invention is not limited to this, and for example, when the image processing server 120 stores the background image in advance or performs image processing without using the background image, the image processing apparatus 100 does not output the background image but only the foreground image. May be output. Further, the image processing device 100 may output the background image at a frame rate lower than that of the foreground image, or may output the background image at a resolution lower than that of the foreground image. This makes it possible to reduce the load on the image processing device 100, the image processing server 120, and the communication path between the image processing device 100 and the image processing server 120. In particular, according to the present embodiment, since only the region of a predetermined subject to be extracted can be extracted as the foreground region, the amount of high-quality foreground image data can be reduced, and as a result, the amount of foreground image data and the background can be reduced. The total amount of image data can be reduced.

Further, in the present embodiment, the case where the foreground background separation is performed so as not to extract the pixel at the pixel position specified as the background change region among the pixels of the captured image as the foreground region has been mainly described. However, it is conceivable that a pixel corresponding to a predetermined subject to be extracted may be included in the specified pixel position, for example, when a part of the display surface of the display 2002-2004 is shielded by the player 5001 in the captured image. .. In such a case, among the pixels included in the specified pixel positions, only the pixels that do not correspond to the predetermined subject may be excluded from the extraction target. Specifically, the area of the shape of a person is detected from the difference area image, the pixels included in the detected area are not excluded, and the pixels included in the specified pixel position among the other pixels are included in the difference area image. A method such as excluding from may be adopted. According to such a method, even when the pixel corresponding to the predetermined subject to be extracted is included in the specified pixel position in the captured image, the image of the predetermined subject can be extracted.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a recording medium readable by a computer.

10 Image processing system 100 Image processing device 110 Imaging device 120 Image processing server

Claims (16)

An image processing device that extracts a subject area corresponding to a predetermined subject from an image based on the image taken by the photographing device.
The imaging device within the second period designated by the second predetermined operation based on the difference between a plurality of images based on the imaging by the imaging device at a plurality of timings within the first period designated by the first predetermined operation. A specific means for specifying an area inside the image to be extracted based on the image taken by the image, which is not the target of extraction of the subject area, and a specific means.
Extraction to extract the subject area inside the extraction target image based on the difference between the extraction target image and another image based on the shooting by the shooting device at a timing different from the shooting timing of the extraction target image. An image processing apparatus comprising: means for extracting the subject area composed of pixels not included in the area specified by the specific means. The extraction means obtains the subject area by removing the area specified by the specific means from the internal area of the extraction target image determined based on the difference between the extraction target image and the other image. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized by extraction. The extraction means includes a partial image of a partial image of the extraction target image and composed of pixels not included in the region specified by the specific means, and a region corresponding to the partial image of the other image. The image processing apparatus according to claim 1, wherein the subject area is extracted by determining the difference. Having an output means for separately outputting an image based on the pixels of the extraction target image, which are not included in the area extracted as the subject area by the extraction means, and the image of the subject area. The image processing apparatus according to any one of claims 1 to 3, which is characterized. The specifying means specifies a region inside the extraction target image at a pixel position where the difference between the pixel values of the corresponding pixels in the plurality of images is equal to or greater than the first threshold value as a region that is not the target of extraction of the subject region. death,
The extraction means is included in a region of the pixels of the extraction target image whose pixel value difference from the corresponding pixel in the other image is equal to or larger than the second threshold value and is specified by the specific means. The image processing apparatus according to any one of claims 1 to 4, wherein an area composed of non-pixels is extracted as the subject area. The predetermined subject is not included in the photographing range of the photographing apparatus in the first period, and the predetermined subject is included in the photographing range of the photographing apparatus in at least a part of the second period. The image processing apparatus according to any one of claims 1 to 5. Any one of claims 1 to 6, wherein the region specified by the specific means is a region different from the predetermined subject and corresponding to a subject that changes with the passage of time. The image processing apparatus described in the section. The subject area extracted by the extraction means is an area corresponding to a person as a predetermined subject.
The image processing apparatus according to claim 7, wherein the area specified by the specific means is an area corresponding to the display surface of the display device as another subject. The other image is an image based on the image taken by the photographing apparatus within the first period.
The image processing apparatus according to any one of claims 1 to 8, wherein the extraction means extracts the subject area by using a background subtraction method. The other image is an image based on the image taken by the photographing apparatus within the second period, and the extraction means is characterized in that the subject area is extracted by using the inter-frame difference method. The image processing apparatus according to any one of the above items. The image processing apparatus according to any one of claims 1 to 10.
The image of the subject area extracted by the extraction means and the image of the area corresponding to the predetermined subject extracted from the image taken by another photographing device that photographs from a direction different from that of the photographing device. Based on this, an image processing system comprising an image generation means for generating a virtual viewpoint image including the predetermined subject. An acquisition means for acquiring viewpoint information according to the designation of the virtual viewpoint related to the generation of the virtual viewpoint image, and
Three-dimensional of the predetermined subject based on the image of the subject area extracted by the extraction means and the image of the area corresponding to the predetermined subject extracted from the image based on the image taken by the other photographing device. It has a model generation means to generate a model, and has
The image generation means is characterized in that it generates a virtual viewpoint image including the predetermined subject based on the viewpoint information acquired by the acquisition means and the three-dimensional model generated by the model generation means. Item 11. The image processing system according to item 11. It is an image processing method executed by a system that extracts a subject area corresponding to a predetermined subject from an image based on an image taken by a photographing device.
The imaging device within the second period designated by the second predetermined operation based on the difference between a plurality of images based on the imaging by the imaging device at a plurality of timings within the first period designated by the first predetermined operation. A specific step of specifying an area inside the image to be extracted based on the image taken by the photographer and not the area to be extracted of the subject area.
Extraction to extract the subject area inside the extraction target image based on the difference between the extraction target image and another image based on the shooting by the shooting device at a timing different from the shooting timing of the extraction target image. An image processing method comprising a means, which comprises an extraction step of extracting the subject region composed of pixels not included in the region specified in the specific step. In the extraction step, the subject area is determined by removing the area specified in the specific step from the internal area of the extraction target image determined based on the difference between the extraction target image and the other image. The image processing method according to claim 13, wherein the image is extracted. In the extraction step, a partial image of the extraction target image, which is composed of pixels not included in the region specified in the specific step, and a region corresponding to the partial image of the other image. The image processing method according to claim 13, wherein the subject area is extracted by determining the difference. A program for operating a computer as each means of the image processing apparatus according to any one of claims 1 to 10.

Images