JP2011229008A - Image capturing apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image capturing apparatus which prevents an identical moving object from coming out in multiple image capturing areas when combining images captured in multiple capturing areas, which are formed by dividing a capturing range, to create a panorama composite image and provides high quality combined images.SOLUTION: A control device controls a camera and an automatic tripod head to capture images in each image capturing area formed by dividing a image capturing range. The control device processes multiple images captured in one image capturing area under different conditions. Then, the control device extracts a moving object 1101 in the image capturing area and presumes future movements of the moving object 1101 in the image capturing range. The control device captures images in image capturing areas which the moving object is unlikely to exist in advance and leaves areas where images have not been captured and the moving object is likely to be captured until later based on the presumption result.

Description

  The present invention relates to an image capturing apparatus that acquires a captured image for forming a composite image of the entire shooting range from individual shooting areas obtained by dividing the shooting range, and more specifically, so that a plurality of the same moving objects are not captured on the combined image. Related to control.

  A system has been put to practical use in which photographic images taken in individual shooting areas obtained by dividing a shooting range are combined to form a photographic image in the shooting range (Patent Documents 1 and 2).

  Japanese Patent Application Laid-Open No. 2004-228561 discloses an image photographing apparatus that automatically photographs a panoramic landscape photograph. Here, when the shooting range is divided and a plurality of shooting areas are set in advance, the support mechanism of the digital camera is controlled, and the angle of view of the digital camera is automatically set in each shooting area to continuously take pictures. To do. In the following, the latter is referred to as an angle of view (so-called camera angle) in order to distinguish between a shooting area set on the subject side and a shooting area targeted from the camera side.

  Patent Document 2 discloses a medical image photographing apparatus that automatically photographs an uneven subject surface. Here, each time a photograph is taken in a photographing area set by dividing the photographing area, the photographs taken are sequentially combined, and a composite image of the photographing range is displayed on the display device.

  In Patent Document 3, the exposure is changed, panning is performed a plurality of times, density gradation is extracted from a white-out image or a black-out image, and the dynamic range of the image density (brightness / darkness) of the final image is expanded. A panoramic shooting method is shown.

  In Patent Document 4, an image before and after a focal position that is out of focus at one focal position is taken in with a different focal position, and is combined with a captured image at the original focal position. A shooting method for increasing the resolution is shown.

JP 2000-324380 A JP-A-9-173298 JP 11-65004 A JP 2009-92892 A

  A panoramic image shooting system that switches the shooting area sequentially by swinging the angle of view of one camera and continuously shooting images in a plurality of shooting areas was prototyped and operated. The same subject was reflected in This is because a moving object (person) shot in one shooting area has been shot again after moving to another shooting area during continuous shooting. And since the panoramic image in which the same subject is reflected in a plurality of places in the photographing range gives an unnatural impression, it is evaluated that the quality is poor.

  Here, as shown in Patent Documents 1 and 3, in the case of synthesizing about 2 to 5 photographic images, since the shooting is completed in a short time, the same moving object is shot in a plurality of shooting areas. Is unlikely. However, in panoramic image shooting in which a large number of shooting areas are taken over a large number of divided areas, the possibility that the same moving object is shot in a plurality of shooting areas increases. Then, by dividing and shooting the entire shooting area and combining it, the image resolution is improved and a high resolution image is created, so that it is obvious at a glance that they are the same moving object on the panorama combined image.

  Also, if there is only one moving object in the shooting range, it can be dealt with by manually changing the shooting order so as to avoid that moving object, but when there are multiple moving objects in the shooting range, It cannot be dealt with by human judgment.

  An object of the present invention is to provide an image photographing apparatus capable of forming a high-quality composite image by preventing a plurality of identical subjects from appearing on the composite image.

  The image photographing apparatus of the present invention includes photographing means capable of photographing individual photographing areas obtained by dividing a photographing range, and moving means capable of positioning the angle of view of the photographing means in different photographing areas by moving the photographing means. The image capturing device and the control device for controlling the moving device are provided so as to acquire a captured image in each image capturing area. The control means detects a moving object that moves in the shooting area based on the acquired captured image, and avoids shooting of the moving object in an unphotographed shooting area based on the detection result of the moving object. The photographing means and the moving means are controlled.

  In the image capturing apparatus of the present invention, since the moving object in the imaging region is detected from the acquired captured image, the position of the moving object in the imaging region is accurately detected, and the movement of the imaging range of the moving object thereafter is accurately detected. Can be estimated. For this reason, by controlling the photographing means and the moving means based on the detection result of the moving object, it is possible to avoid with high accuracy that the same moving object is photographed again in the unphotographed photographing area.

  Therefore, it is possible to prevent a plurality of the same subject from appearing on the composite image and form a high-quality composite image.

It is explanatory drawing of a structure of a panoramic imaging device. It is explanatory drawing of a camera holder. It is explanatory drawing of the coordinate system of an automatic camera platform. It is explanatory drawing of the distribution of the to-be-photographed object and moving object in an imaging range. It is explanatory drawing of the multiple reflection of a moving object. It is explanatory drawing of a setting screen. It is explanatory drawing of an example of an imaging | photography schedule. It is a flowchart of the program which calculates | requires the focus position used for a depth-of-field synthesis | combination. It is explanatory drawing of the focus position used for a depth-of-field composition. It is explanatory drawing of the picked-up image for dynamic range expansion. It is explanatory drawing of the estimation method of the movement of a moving object. 3 is a flowchart of panoramic shooting control according to the first embodiment. It is explanatory drawing of a warning screen. 10 is a flowchart of panorama shooting control according to the second embodiment. It is explanatory drawing of a standby screen. FIG. 10 is an explanatory diagram of a shooting schedule in panoramic shooting control of Embodiment 3. 12 is a flowchart of panoramic shooting control according to a fourth embodiment. It is explanatory drawing of the picked-up image in which the moving object was reflected. FIG. 10 is an explanatory diagram of a method for determining multiple reflections in Embodiment 4.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The present invention is also implemented in another embodiment in which part or all of the configuration of the embodiment is replaced with the alternative configuration as long as the automatic continuous shooting of the shooting range is performed so as to avoid the reflection of the moving object. it can.

  In addition, about the general matter regarding the imaging | photography apparatus shown by patent documents 1-4, and the synthesis | combination process of a panoramic image, illustration is abbreviate | omitted and the overlapping description is abbreviate | omitted.

<Image shooting device>
FIG. 1 is an explanatory diagram of a configuration of a panoramic imaging apparatus. FIG. 2 is an explanatory diagram of the camera holder. FIG. 3 is an explanatory diagram of the coordinate system of the automatic camera platform. As shown in FIG. 1, the panoramic imaging apparatus 100 is configured to capture an imaging area obtained by dividing the imaging range by moving the angle of view of the camera 101 fixed to the camera holder 102 in the vertical direction and the horizontal direction by the automatic camera platform 103. Position to any one. The panorama imaging device 100 performs shooting by positioning the angle of view of the camera 101 at each shooting position, and displays the acquired shot image on the display device 105.

  A camera 101 which is an example of a photographing unit can photograph individual photographing areas obtained by dividing a photographing range. The camera 101 photographs a subject. As the camera 101, a commercially available digital single-lens camera is used, and an interchangeable lens having a known focal length and F value is connected in advance. Any control related to shooting and shooting conditions such as focus control, aperture value control, exposure time control, shutter control, and image transmission of the camera 101 can be controlled by communication with the control device 106.

  The camera holder 102 fixes the camera 101 and the lens stably. The camera holder 102 has a function of setting the position of the camera 101 at predetermined reference positions in the pan direction, the tilt direction, and the rotation direction.

  As shown in FIG. 2, a camera installation reference line 201 is drawn on the bottom surface of the camera holder 102. By fixing the camera 101 with reference to the camera installation reference line 201, the camera 101 can be mounted so as to face the camera holder 102. The camera holder 102 is provided with a level 202. After the camera 101 is fixed, the level (rotation angle and tilt angle) of the camera 101 can be adjusted to 0 degrees with reference to the level 202.

  A laser pointer 203 is mounted on an extension line of the camera installation reference line 201. By directing the laser beam of the laser pointer 203 to a predetermined position of the subject 204, the orientation of the camera 101 can be accurately set.

  An automatic camera platform 103 that is an example of a moving unit can move the camera 101 and position the angle of view of the camera 101 in different imaging regions. The automatic camera platform 103 is connected to the camera holder 102 to perform a pan / tilt operation in order to control the shooting direction of the camera 101. The tripod 104 installs the camera 101 at a predetermined height. As shown in FIG. 1B, the automatic camera platform 103 can pan and tilt the camera 101 by operating the pan mechanism 111 and the tilt mechanism 112 in accordance with an instruction from the control device 106.

  As shown in FIG. 3A, the entire coordinate system with the camera 101 as the origin is set. First, the camera 101 is leveled using the level 202 shown in FIG. Next, the camera 101 is panned while irradiating the subject 204 with the laser pointer 203, and the Y-axis is defined in the depth direction by aligning the front direction of the camera with the left and right centers of the imaging region. Finally, the Y-axis orthogonal axis in the horizontal plane is defined as the X-axis, and the vertically upward direction is defined as the Z-axis. Thereby, the orthogonal coordinates XYZ are defined. In the pan / tilt control of the automatic camera platform 103, the panning angle is defined as an angle φ from the Y axis as shown in FIG. 3B, and the tilt angle is defined from the XY axis plane as shown in FIG. Is defined by the elevation angle θ.

  The control device 106, which is an example of a control unit, controls the camera 101 and the automatic camera platform 103 so as to acquire a photographed image in each photographing region. A control device 106, which is an example of a computer, is equipped with a program that controls the camera 101 and the automatic camera platform 103 to acquire captured images in a plurality of imaging regions. In the present embodiment, the control device 106 uses a single personal computer for the convenience of carrying. However, the control device is divided into a plurality of arithmetic devices, control modules, image processing devices, and the like. May be.

  The control device 106 captures the setting of the shooting conditions of the camera 101 through the input unit 108, performs pan / tilt control of the automatic camera platform 103, performs shooting control of the camera 101, and displays the acquired shot image on the display device 105. Image processing. The control device 106 controls the camera 101 and the automatic pan / tilt head 103 regarding photographing, acquisition of a photographed image from the camera 101, image processing of the acquired image, and output of the image processing result to the display device 105. For shooting, a control signal is output so as to perform pan / tilt control of the automatic camera platform 103, focus adjustment, aperture value setting, shutter time setting, and release according to a flow described later.

  The display device 105, which is an example of a display unit, displays the captured image acquired in the imaging area in association with the imaging range. The display device 105 reduces the captured image acquired in each imaging region and displays the images in an arrangement corresponding to the imaging range. FIG. 1 shows a state in which photographing of a 20-divided panorama divided image 107 with 5 divisions in the pan direction and 4 divisions in the tilt direction has been completed. The image is additionally displayed in the order in which the reduced images are acquired, with the tilt direction of the shooting range being vertical and the pan direction being horizontal.

  In panoramic image shooting, a wide range of the subject area exceeding the angle of view determined by the optical system specifications of the digital camera is shot. The shooting range is divided into continuous or partially overlapping shooting areas less than the angle of view of the digital camera, and each of the divided shooting areas is shot with the digital camera. Then, the photographed images in the photographing regions that have been photographed and recorded are connected and connected (stitched) by image processing to reproduce a photographic image in a wide photographing range.

<Multiple moving objects>
FIG. 4 is an explanatory diagram of the distribution of the subject and the moving object in the shooting range. FIG. 5 is an explanatory diagram of multiple reflection of a moving object. As illustrated in FIG. 4, the panoramic imaging apparatus 100 repeats shooting while changing the direction of the camera 101 and moving the angle of view for each shooting region. Since the panorama imaging apparatus 100 captures a wide range of areas such as landscapes, the panorama imaging apparatus 100 is often photographed outdoors, and various moving objects 2102 such as people, cars, animals, and natural objects exist in the imaging range 2101.

  Here, image blurring in a captured image caused by movement of a moving object existing in an individual imaging region can be corrected later by applying a conventional blur correction process to the captured image.

  However, if the moving direction of the moving object matches the visual field moving direction of panoramic shooting and the apparent moving speed of the moving object is close to the visual field moving speed of panoramic shooting, the same moving object appears in the captured images of multiple shooting areas. Problem occurs.

  As shown in FIG. 5, a panoramic image in which the same moving object is reflected in a plurality of divided images is unnatural at first glance. Such multiple reflection of the same object is a factor that greatly impairs the image quality of the panoramic image and is a serious problem.

  Conventionally, after synthesizing panoramic images, it has been known for the first time that multiple reflections exist in which the same object is present at different locations, and it takes time and effort to redo the time-consuming shooting from the preparation stage. The problem of multiple reflection is a problem that can always occur when a panoramic image is divided and photographed. And in the panoramic imaging device 100, as will be described later, in order to acquire a plurality of captured images in the same imaging region, it will take a lot of time from the start to the end of shooting, The possibility of multiple reflections increases.

  For this reason, in the panorama imaging apparatus 100, the control device 106 detects a moving object that moves in the shooting area based on the acquired plurality of captured images, and moves in the unphotographed shooting area based on the detection result of the moving object. Avoid shooting objects. The control device 106 shoots a plurality of continuous shooting areas according to a predetermined shooting schedule, and processes the acquired shot images and predicts the movement of the moving object when a moving object is detected in the shooting area. Then, based on the estimation result of the movement of the moving object, the camera 101 and the automatic camera platform 103 are controlled to move the moving object to one unphotographed area or after the moving object passes through one unphotographed area. A photographed image is acquired in the unphotographed area. As a result, it is possible to obtain a panoramic composite image without multiple reflections in a short time.

<Shooting schedule>
FIG. 6 is an explanatory diagram of a setting screen. FIG. 7 is an explanatory diagram of an example of a shooting schedule. FIG. 8 is a flowchart of a program for obtaining a focal position used for depth of field synthesis.

  As shown in FIG. 6 with reference to FIG. 1, the control device 106 allows the user to input shooting conditions through the setting screen 120 displayed on the display device 105 by the panorama shooting program. When an input is made through the operation unit 108 or the like, the control device 106 sets a shooting schedule according to the input content.

  As shown in FIG. 6, the number of divisions of the shooting range for panoramic shooting (pan direction / tilt direction), the rotation angle (pan angle / tilt angle) of the automatic camera platform for each shooting region (small field of view), and rotation priority The direction (pan or tilt) is input.

  When the setting is made, the control device 106 determines a shooting schedule by calculating a shooting region obtained by dividing the shooting range as shown in FIG. As shown in FIG. 8, a calculation program is executed to determine a pan angle P [k] and a tilt angle [k] for setting each imaging region in the camera 101. The calculation processing flow of FIG.

  Expression (1) is a program example for obtaining the pan angle φ and tilt angle θ shown in FIG. Here, from the number of divisions in the pan direction and the pan head rotation angle (N: 5, p: 20.4) and the number of divisions in the tilt direction and the pan head rotation angle (m: 4, T: 13.7) in FIG. , The pan angle φ and the tilt angle θ corresponding to each imaging region for performing 20-division imaging are obtained. As shown in FIG. 7, the pan angle P [k] and the tilt angle [k] are calculated using the frame number k in the shooting order for 20 shooting regions as an index.

As shown in FIG. 7, the shooting start point 802 is a position panned 40.8 degrees to the left and 20.6 degrees tilted upward with respect to the coordinate origin 803. (N, p) is the number of divisions in the pan direction (N: 5) and the pan head rotation angle (p: 20.4) set in FIG. (M, T) is the number of divisions in the tilt direction (m: 4) and the pan head rotation angle (T: 13.7) per division set in FIG.

  The control device 106 performs automatic continuous shooting from the shooting start point 802 as a starting point according to the set shooting schedule. Continuous shooting is executed while moving the shooting area on the automatic camera platform 103 with priority given to the pan direction (horizontal direction). The control device 106 controls the camera 101 and the automatic camera platform 103 so as to acquire a plurality of photographed images with different photographing conditions in one photographing region.

<Dynamic range expansion, depth of field composition>
FIG. 9 is an explanatory diagram of the focal position used for the depth-of-field composition. FIG. 10 is an explanatory diagram of a captured image for dynamic range expansion.

  In panoramic landscape photography, a wide range of field of view is taken.Therefore, the brightness difference between bright and dark places in the shooting range is larger than in normal subjects. Black spots tend to occur in dark areas. If the image density range is assigned to the entire brightness range of the shooting range in order to avoid blackouts and blackouts, the number of gradations is insufficient in each part, resulting in poor stereoscopic effect and smooth gradation. It will be missing.

  For this reason, in the panorama imaging apparatus 100, a dynamic range of gradation expression of a panoramic image and smoothness of gradation are obtained by capturing and synthesizing a plurality of images with different aperture values in a plurality of stages in each photographing region. Is increasing.

  Further, in the panorama imaging apparatus 100, in order to increase the resolution of the panorama composite image, the shooting area is reduced using a lens having a narrow viewing angle and a long focal length, thereby increasing the resolution for each shooting area. Yes. However, when shooting using a lens with a long focal length, the depth of focus is inevitably shallow, and a large number of images with a reduced focus and reduced resolution appear in the captured image.

  For this reason, the panoramic imaging apparatus 100 captures a plurality of photographed images by changing the focal position of the lens in a plurality of stages in one photographing region. By performing depth-of-field synthesis that synthesizes photographic images with different focal positions, a synthesized image that is focused on every corner of the photographic region can be obtained. In the depth-of-field combination, only the in-focus portions of the captured images are connected.

  As shown in FIG. 6, in order to perform dynamic range expansion synthesis and depth-of-field synthesis, a shooting exposure amount (aperture value, shutter speed, ISO sensitivity) is set on the setting screen 120. Then, prioritization of each setting item is set. Here, as an example, multiple-photographing with different exposure amounts for expanding the dynamic range is given first priority, and multiple-photographing at different focal positions for depth-of-field composition is given second priority.

  As shown in FIG. 9 with reference to FIG. 1, a plurality of focus positions (focus positions) for depth-of-field synthesis are set as follows. The depth of field is a calculable numerical value that is uniquely determined by the focal length (f value), aperture (F value), and subject distance (focal position) of the lens. FIG. 9 shows a focal position (focus distance) and a focal range at the time of the diaphragm F16 in each of the focal lengths of 85 mm, 100 mm, 135 mm, and 300 mm.

  When photographing with a diaphragm F16 using a lens having a focal length of 85 mm, when the focal position (focus ring) is set to 4.6 m, the in-focus range is 3.4 m to 6.9 m. When the focal position is set to 14 m, the focusing range is 6.9 m to infinity. Therefore, two images are shot with the focus position (focus ring) set to 4.6 m and 14 m, and depth-of-field composition processing is performed, so that the composition is in focus at an object distance of 3.4 m to infinity. You can create an image.

  When shooting with a diaphragm F16 using a lens with a focal length of 300 mm, if the focal position (focus ring) is set to 20 m, the in-focus range is 18 m to 22 m. When the focal position (focus ring) is set to 25 m, the focusing range is 22 m to 29 m. When the focal position (focus ring) is set to 35 m, the focusing range is 29 m to 44 m. When the focal position (focus ring) is set to 60 m, the focusing range is 44 m to 93 m. Furthermore, when the focal position (focus ring) is set to 170 m, the focusing range is 84 m to infinity. Therefore, with a lens with a focal length of 300 mm, five images with different focal positions (focus rings) are taken in five stages, and depth-of-field composition processing is performed, so that a subject distance of 18 m to infinity is in focus. Combined composite images can be created.

  As illustrated in FIG. 10 with reference to FIG. 1, the control device 106 acquires three captured images continuously by changing the shutter speed in three stages in one capturing area. This is because the dynamic range of the image density is expanded by synthesizing photographed images of the same subject with different exposure amount settings. Then, the control device 106 detects a moving object in the shooting area using the shot images having different exposure amounts in three stages.

<Moving object detection>
When the photographing of all exposure levels is completed in one photographing region, the control device 106 performs moving object detection by performing image processing (calculation processing) on a plurality of photographed images having different exposure levels in a plurality of stages. The control device 106 extracts a moving object in the imaging region based on a plurality of captured images obtained by different imaging conditions acquired in one imaging region, and estimates subsequent movement in the entire imaging range. When shooting with the exposure level changed in three stages is completed, the control device 106 processes three images with different exposure levels and detects a moving object in the shot image. And the control apparatus 106 acquires a picked-up image in the imaging area | region where a moving object does not exist by controlling the camera 101 and the automatic camera platform 103 based on the estimation result of the movement of a moving object.

As shown in FIG. 10, photographed images with different exposure levels are called a low-exposure photographed image 1001, a reference-exposure photographed image 1002, and a high-exposure photographed image 1003 in order from the smallest exposure amount. _Are P _low , P ₀ , and P _high .

  Since the captured images 1001, 1002, and 1003 have different brightnesses, the brightness is normalized by the following expression to obtain a low-exposure normalized image 1004, a reference-exposure normalized image 1005, and a high-exposure normalized image 1006. Convert.

In Expression (2), the image signals of the normalized images 1004, 1005, and 1006 whose brightness has been normalized are P _low ′, P ₀ ′, and P _high ′, respectively. S _low , S ₀ , and S _high are the sum of the image signal values in each screen.

  The captured image with the normalized brightness level is subjected to template matching between the low-exposure normalized image 1004 and the high-exposure normalized image 1006 to calculate a moving object motion vector v on the captured image. The direction and speed of the moving object are combined to define a “moving object motion vector v”. Further, the moving direction and moving speed of the shooting area are combined to define a “shooting field motion vector”.

  By subtracting the exposure levels of the captured images taken after a certain amount of time, the still area and the moving area in the captured image can be separated, and only the moving area is extracted.

  The extracted moving area is used as a moving object template for template matching. The moving object template is shifted from the position where the moving object template and another image are overlapped as the origin, and the shift amount to the position coordinate where the subtraction value between the moving object template and another image is the minimum is the movement amount of the shooting interval. It is defined as By dividing the moving amount by the shooting interval, the moving speed of the moving object in the shooting region can be obtained.

  The detection of the moving object in the shooting area is not limited to the method using the shot images having different exposure amounts in three stages. In the moving object detection, an image may be taken into the camera 101 as a moving image at the time of preview before the shutter operation, and the moving object may be detected.

<Estimation of moving object>
FIG. 11 is an explanatory diagram of a method for estimating movement of a moving object. As shown in FIG. 11 with reference to FIG. 1, when the moving object 1101 is detected in the captured image, the control device 106 performs multiple reflection prediction of the moving object in the uncaptured imaging region. When the moving object 1101 is detected, the control device 106 predicts whether or not the moving object moves to an unphotographed imaging area and is imaged again.

  FIG. 11 is a diagram showing the entire panoramic image, and shows a state of arrangement of divided images of 5 divisions in the pan direction and 4 divisions in the tilt direction. FIG. 11 shows a state where shooting is started from the upper left and shooting of the second divided image in the pan direction is completed, and the divided images divided by broken lines in the drawing represent divided images to be shot from now.

  Here, since the rotation direction of the automatic camera platform 103 at the time of shooting is prioritized in the pan direction, the description is limited to the movement of a moving object in the horizontal direction. However, when the rotation direction of the automatic camera platform 103 has priority on the tilt direction, it can be similarly obtained by replacing all the parameters described below with the vertical direction.

In the second divided image, it is assumed that the moving object 1101 having the horizontal speed v is detected at the position x of the horizontal pixel number L by the moving object detection of FIG. The moving object 1101 appears in the next divided image when the object moved at the speed v is in the third divided image (positions A to B in FIG. 11 represent both ends of the image) and the shutter is released. It is. The time T for moving the object at the position x to the position A at the speed v is expressed by the following equation.
T = (L−x) v

Further, the time T for moving to the position B after traveling the distance L is expressed by the following equation.
T = (2L−x) v

  For this reason, when the interval time between the divided captured images is T, the prediction formula for the condition that the moving object is reflected in the next divided captured image is as follows.

  When the moving object satisfies Expression (3), the control device 106 interrupts the normal shooting schedule and executes various additional processes based on the prediction result of the multiple reflection prediction.

<Example 1>
FIG. 12 is a flowchart of panoramic shooting control according to the first embodiment. FIG. 13 is an explanatory diagram of a warning screen. In the first embodiment, it is determined whether the moving object is captured in another divided image from the predicted movement of the moving object and the shooting schedule.

  As illustrated in FIG. 12 with reference to FIG. 1, in the first embodiment, the control device 106 displays a display device when there is an unphotographed area where there is a possibility of photographing a moving object based on the detection result of the moving object. A warning is output on 105. When there is a high possibility that the moving object is photographed again in the unphotographed imaging area due to the estimation of the movement of the moving object, a warning is displayed on the display device 105.

  When an instruction to start shooting is input, the panorama imaging apparatus 100 starts panorama automatic shooting (S601). Specifically, the user presses the start button displayed on the display device 105 by the panorama shooting program with the mouse.

  When shooting is started, the system is first initialized (S602). In the initialization, as shown in FIG. 3A, the automatic camera platform 103 moves to the origin coordinates, and at the same time, as shown in FIG. Hit a subject. Thereby, the user can confirm the origin coordinates of the automatic camera platform 103 visually. Thereafter, the shooting schedule determined based on the setting contents of FIG. 6 is read to initialize the shooting schedule.

  When the initialization is completed, the field of view setting (S603) is executed, and the automatic camera platform 103 pans and tilts to position the imaging region at the imaging start point. When the coordinate origin 803 in FIG. 7 is initially set, a shooting area at the upper left corner of the panorama image shooting range is set as a shooting start point 802.

  When the field of view setting is completed, focus setting (S604) is executed, and the focus position in FIG. 6 is set. In the first embodiment, a lens having a focal length of 100 mm is used at the diaphragm F16, and the subject distance range is 3.3 m to infinity. Therefore, using the setting of f100 in FIG. 9, the focal position is set to 19 m, 6.7 m, and 4 m from a distance, and three shot images are acquired in one shooting area. In the first shooting, a focus ring is set at the focal position 19m (S604).

When focus setting is completed, exposure setting (S605) is executed to set the aperture value, shutter speed, and ISO sensitivity. In the first embodiment, the exposure level for dynamic range expansion processing is photographed with three types of exposure amounts of 1/2 times, 1 times, and 2 times by changing the shutter speed. Value F16
The shutter speed was set to 1/500 seconds and the ISO sensitivity was set to 100. The exposure amount can be set with any of aperture value, shutter speed, and ISO sensitivity. However, changing the aperture value changes the depth of field, which affects the shooting conditions for combining depth of field. Is not preferable.

  When the field of view setting (S603), focus setting (S604), and exposure setting (S605) are completed, the first shooting in the shooting region at the shooting start point 802 is executed (S606).

  For shooting, the shutter is released in accordance with an instruction from the control device 106, and the captured image is immediately transmitted from the camera 101 to the control device 106. When shooting is completed, it is determined whether shooting of all setting items with the highest priority is completed (S607). In the first embodiment, the first priority is the exposure amount setting, and it is determined whether or not the photographing at the shutter speeds of 1/500 seconds, 1/250 seconds, and 1/125 seconds is completed.

  When shooting with the exposure level changed in three stages is completed, the control device 106 processes three images with different exposure levels using the procedure described with reference to FIG. An object is detected (S608).

  Then, the control device 106 uses the procedure described with reference to FIG. 11 to predict whether or not the moving object is reflected in other shooting areas in the panoramic shooting based on the detection result of the moving object. Prediction (S609) is executed.

  When the possibility of multiple reflection of a moving object is detected in the multiple reflection prediction (S609), a warning is issued to the display device 105, automatic continuous shooting is interrupted, and a user operation is waited for. As a warning method, it is desirable to notify the user of the possibility of moving object detection and multiple reflection by voice.

  As illustrated in FIG. 13 with reference to FIG. 1, the control device 106 performs display based on the detection result of the moving object on the display device 105 in association with the imaging range. In FIG. 13, (a) and (b) show examples of warnings by the display device 105.

  As shown in FIG. 13A, when a reduced image of a captured image area is displayed on the display device 105 and a possibility of multiple reflection of a moving object is predicted, a warning dialog is displayed on the screen together with sound. 1201 is displayed. The reduced image of the shooting area in which the moving object is detected draws attention such as blinking of the image frame, and when the mouse pointer is applied, an enlarged image is displayed as shown in FIG. The moving object detected in S608) can be confirmed.

  The user waits until the moving object passes through the next photographed image, and then presses the resume button 1202 to continue photographing. The user may end shooting by pressing the application end button 1203.

  When shooting is resumed by the photographer's operation (S611), it is determined whether shooting with the focus position changed is completed (S612). Whether moving object detection (S608) does not detect a moving object or multiple reflection prediction (S609) does not predict multiple reflections of a moving object, whether or not shooting with a changed focus position is completed (S612).

  When shooting with the focal position changed according to the shooting schedule is completed (YES in S612), the automatic camera platform 103 is activated to start shooting in the next shooting area (S603). However, if shooting with the focus position changed is not completed (NO in S612), the process returns to the focus setting (S604), and the next focus position is set and shooting is continued.

  When exposure control shooting and focus control shooting are completed (YES in S612), it is determined whether or not full-angle shooting has been completed (S613). If shooting with all field angles has not been completed, the next field of view is set (S603) and shooting is continued. When the all-division shooting is completed (YES in S613), the shooting is ended (S614).

  In the first embodiment, a moving object being photographed is detected, and it is predicted whether or not the same moving object is photographed even in another photographing region. If the same moving object is predicted to be shot in another shooting area, a warning is issued, automatic shooting is interrupted, and the operation is returned to the photographer to reflect the moving object in the other shooting area. To avoid.

<Example 2>
FIG. 14 is a flowchart of panoramic shooting control according to the second embodiment. FIG. 15 is an explanatory diagram of a standby screen. In the second embodiment, the occurrence of multiple reflections of moving objects in the panoramic image imaging system is determined, the shooting schedule is changed, and control is performed so as to avoid multiple reflections.

  The second embodiment is the same as the first embodiment except that it waits for a moving object to pass through the shooting area and starts shooting. Therefore, the control steps common to the first embodiment in FIG. Common reference numerals are assigned and redundant explanations are omitted.

  As illustrated in FIG. 14 with reference to FIG. 1, in the second embodiment, the control device 106 acquires captured images in a certain order for a plurality of captured areas. The control device 106 waits until the moving object passes based on the estimation result of the movement of the moving object, and acquires the captured image for the imaging area determined based on the estimation result of the movement of the moving object. To do. The second shooting of the moving object is avoided by interrupting the shooting up to the shooting area immediately before the shooting area where the moving object exists, and waiting.

  In the second embodiment, a timer (S1301) is installed in a program executed by the control device 106. The control device 106 processes a plurality of captured images to detect a moving object (S608), obtains a moving vector of the moving object, and performs multiple reflection prediction (S609). If it is determined that there is a possibility that the moving object will appear, a warning is issued and at the same time, the shooting is interrupted (S611), and the timer counts down the time described below (S1301).

As shown in FIG. 11, the moving object 1101 having the velocity v existing at the position x does not appear after passing through the next imaging region when the time T of the following equation elapses.
T> (2L−x) / v

Accordingly, the timer (S1301) counts down the time T sequentially starting from the shooting interruption start time, and starts shooting the next shooting area after the time T of the following expression has elapsed (S1612).
T = (2L−x) / v

  As shown in FIG. 15, during the countdown of the timer (S1301), a warning dialog 1401 is displayed on the display device 105 (S611). Through the warning dialog 1401, a time until the resumption of photographing is displayed together with a warning of multiple reflections of moving objects. When permitting the reflection of the moving object, the user presses the resume button 1202 to continue the shooting. Further, the user may end shooting by pressing the application end button 1203.

  According to the panorama shooting control of the second embodiment, the user can immediately resume shooting without operating the panoramic imaging device 100 when the moving object goes out of the next shooting area. Shooting is completed in a shorter time.

<Example 3>
FIG. 16 is an explanatory diagram of a shooting schedule in the panoramic shooting control of the third embodiment. In the third embodiment, a shooting area where a moving object is estimated to be skipped is shot, and a shooting area where no moving object is present is shot in advance, and shooting of a shooting area where a moving object is estimated is postponed.

  As shown in FIG. 12 with reference to FIG. 1, the control device 106 extracts a moving object in the shooting area based on a plurality of shot images shot in one shooting area. The control device 106 estimates the subsequent movement of the extracted moving object in the shooting range, and controls the camera 101 and the automatic camera platform 103 so as to acquire a shot image in a shooting area where no moving object exists based on the estimation result. To do.

  As shown in FIG. 7, a shooting area is set in the shooting range and a shooting schedule is set. As described above, the pan angle P [k] and tilt angle [k] of each shooting area are determined, and the shooting schedule is set. Arranged in order.

  As shown in FIG. 1, the control device 106 reads the shooting schedule list in Table 1, sets the pan angle P [k] and the tilt angle [k], and sequentially increments from k = 1 to k = 20. Until then, shooting in each shooting area is executed.

  As shown in FIG. 16, for example, when a moving object is detected in a shooting frame of k = 2 (S608) and multiple reflections in a shooting frame of k = 3 are predicted (S609), k = 3. The shot frame is transferred to the end of the shooting schedule list. Then, the shooting schedule list is updated so as to carry data of k = 4 to k = 20.

  In Table 2, the pan angle P [k] and tilt angle [k] of the shooting frame of k = 3 to k = 19 are the pan angle P [k] of the shooting frame of k = 4 to k = 19 in Table 1 before the change. k] and tilt angle [k]. Then, the pan angle P [k] and tilt angle [k] of the last shooting frame of k = 20 are the pan angle P [k] and tilt angle [k] of the shooting frame of k = 3 in Table 1 before the change. Has been replaced.

  The control device 106 sets the pan angle P [k] and the tilt angle [k] according to the shooting schedule list in Table 2, and executes shooting in an unshooted shooting area. As a result, the shooting schedule is changed as shown in FIG. When it is determined that there is a possibility that the moving object 1101 is reflected in the next shooting area, as shown in the shooting order 1601, shooting in the shooting area where there is a possibility of shooting is skipped, and finally the shooting is performed. Return to shooting in the skipped shooting area and shoot.

  According to the third embodiment, since the continuous shooting of the remaining shooting area is continued without waiting for the moving object 1101 to move, the entire shooting range can be shot in a shorter time than the second embodiment while avoiding the reflection of the moving object. Can be terminated.

<Example 4>
In the fourth embodiment, unlike the first to third embodiments, the moving object detection during continuous shooting is not performed. First, continuous shooting is performed for all shooting areas to obtain image shooting, and after the shooting is finished, moving object detection is performed in all shooting areas, and whether or not the same moving object is shot in two or more shooting areas. Determine whether. Then, only the imaging region where the same moving object is imaged is imaged again later.

  FIG. 17 is a flowchart of panoramic shooting control according to the fourth embodiment. FIG. 18 is an explanatory diagram of a captured image in which a moving object is reflected. FIG. 19 is an explanatory diagram of a method for determining multiple reflections in the fourth embodiment. In the fourth embodiment, re-shooting is performed on a shooting area where multiple reflection has occurred.

  In the fourth embodiment, as in the first embodiment, continuous shooting is performed by sequentially changing the shooting area in accordance with the shooting schedule. Therefore, in FIG. 14, the same reference numerals as those in FIG. A duplicate description is omitted.

  As shown in FIG. 17 with reference to FIG. 1, the control device 106 detects a moving object that is the same as a moving object that has been detected in another imaging area, in a captured image of one imaging area. Then, the photographed image is reacquired in the one photographing region. The control device 106 performs processing for detecting the same moving object in a plurality of captured images, and acquires a captured image again in the imaging region where the same moving object is detected.

  The control device 106 changes the shutter speed (S607), the focal position (S611), and the shooting area (S612), and executes continuous shooting in all the shooting areas (S602) set in the shooting range (S606). After the end of continuous shooting, the control device 106 displays reduced images of all shooting regions on the display device 105 as shown in FIG. At this time, it is assumed that the same moving object 1801 is reflected in the three shooting areas of the shooting range.

  After the end of continuous shooting, the control device 106 detects moving objects in the shot images of all shooting regions by generating motion vectors using the template matching described above (S1701). If no moving object is detected in all imaging regions (non-detection in S1702), the entire imaging process is terminated. Further, even if a moving object is detected (detection in S1702), if the same moving object is not reflected in another imaging area (non-detection in S1704), the entire imaging process is terminated.

  However, as shown in FIG. 18, when a moving object is detected in a plurality of imaging regions (detection in S1702), it is determined whether the moving objects detected in each imaging region correspond to the same object (S1703). ).

  Referring to FIG. 19, an example of a method for determining multiple reflections is shown. FIG. 19 schematically shows FIG. Assume that the shooting schedule of Table 3 is set in the same manner as in the third embodiment, and continuous shooting is performed with shooting frames of k = 1 to k = 20.

  As a result, as shown in FIG. 19, the moving object 1101 at the speed v is reflected in the other photographing areas A, B, and C in a multiple manner to form a multiple image 1901. Here, let L be the length of the imaging region along the moving direction of the moving object 1101. When a moving object 1101 at a speed v at a position x on a straight line of length L is continuously photographed at intervals of time T, the coordinates of positions A, B, and C are N = 1, 2, 3,. It is expressed by the following formula.

  In the multiple reflection determination (S1703), the color, shape, and the like are extracted from the positions A, B, and C of the captured image, and template matching of the moving object 1101 is performed to determine whether or not the moving object 1101 is the same object. I do. If the control device 106 determines that the same object as the moving object 1101 exists at the positions A, B, and C (detection in S1704), it executes re-shooting scheduling (S1705).

  As shown in Table 4, a re-shooting schedule is set for the shooting frames indicated by the thick frames in Table 3. The pan angle P [k] and tilt angle [k] are extracted from the shooting schedules of k = 3, 4, 5 for which multiple reflections are determined, and the re-shooting schedule shown in Table 4 is newly created.

  When scheduling of the re-shooting schedule is completed (S1705), the original shooting schedule is replaced (S602), and re-shooting is started (S603).

  According to the fourth embodiment, the user automatically detects the same moving object even in a panoramic image in which moving objects are reflected in multiple, extracts only a necessary shooting area, and completes shooting in a short time. It becomes possible.

  Note that the control of the fourth embodiment and the control of the second embodiment may be executed in combination. This is because using a plurality of moving object imaging avoidance programs can more reliably prevent multiple reflections of the same moving object.

<Example 5>
In Example 4, continuous shooting was performed for all shooting regions to acquire all image shootings, and then a re-shooting schedule was determined. On the other hand, in the fifth embodiment, during panorama continuous shooting, a process of detecting a moving object that has already been detected from the captured image is performed in parallel, and the same moving object that has been detected in the shooting area that has been captured so far is detected. If detected, they are sequentially added to the end of the existing schedule.

  As shown in FIG. 16, the angle of view is sequentially switched in the order shown in Table 5 to continue automatic shooting, and as shown in FIG. 10, a moving object is detected using the shot image. Here, it is assumed that k is an identification number unique to the imaging region determined along the imaging schedule of FIG.

  As shown in Table 5, it is assumed that the same moving object is detected again in the imaging region of k = 7 after the moving object is detected in the imaging region of k = 2. At this time, the re-shooting schedule for the shooting area of k = 7 is added to the end of the shooting schedule. After that, it is assumed that the same moving object detected in the shooting area of k = 2 is detected again in the shooting area of k = 12. At this time, the reshooting schedule of k = 12 is added after the reshooting schedule of k = 7.

  In the fifth embodiment, a moving object is detected from a captured image in parallel with automatic shooting, and when the same moving object is detected in different shooting areas, the re-shooting schedule is automatically set without waiting for the user's judgment. Add and finish all necessary shooting. For this reason, it can be used in an unmanned imaging system.

  Note that the control of the fifth embodiment and the control of the fourth embodiment may be executed in combination. The control of the fifth embodiment and the control of the second embodiment may be executed in combination. This is because using a plurality of moving object imaging avoidance programs can more reliably prevent multiple reflections of the same moving object.

<Example 6>
In Examples 1 to 5, a moving object is detected using a captured image of the camera 101 for capturing a panoramic divided image. This is because the position and speed of a moving object in a narrow imaging region by a lens with a long focal length can be accurately detected by processing the captured image. If the position and speed of the moving object in the shooting area cannot be accurately detected, the subsequent movement of the moving object in the shooting range outside the shooting area cannot be accurately estimated, and the reflection of the same moving object is avoided. This is because the purpose cannot be achieved.

  On the other hand, a sensor using ultrasonic reflection or laser reflected light, which is known to detect a moving object, is not related to the angle of view range of the camera 101. The possibility of detection increases. For this reason, it is difficult to accurately estimate the movement of the moving object in the imaging range. In addition, a sensor that can detect a moving object with high accuracy in a limited area is generally expensive and large-sized, so that it is not practical as a panoramic photographing system.

  In Examples 1 to 5, a high-resolution captured image used for panoramic synthesis was used as a captured image of the camera 101. By using a high-resolution captured image, it is possible to detect the position and speed of a moving object in the imaging region with high accuracy, and this can improve the estimation accuracy of the subsequent movement of the moving object in the imaging range. It is.

  On the other hand, in the sixth embodiment, a low resolution moving image captured by the camera 101 is used as a preview image before shooting. This is also because it is a plurality of captured images acquired in time series from the imaging region by the camera 101.

  In the sixth embodiment, in each shooting area, the camera 105 captures a preview moving image of the shooting area and transfers it to the control device 106 for 1 second before the actual shooting. The control device 106 reproduces a plurality of captured images arranged in time series from the transferred moving image data, and executes the above-described moving object detection. When a moving object is detected from the moving image data, the above-described estimation of the movement of the moving object is executed, and shooting that avoids the shooting area estimated to be moving is executed.

  According to the panoramic image manufacturing method of the sixth embodiment, as long as the same number of photographed images are used as compared with the case where the final photographed image used for panorama synthesis is used, the position of the moving object is determined. Speed detection accuracy decreases. However, since the amount of data of one photographed image is small, the above-mentioned moving object detection can be executed using several tens of photographed images, which is significantly more than when the final photographed image is used. Speed detection accuracy can be increased to the same level.

DESCRIPTION OF SYMBOLS 101 Camera, 102 Camera holder, 103 Automatic pan head 104 Tripod, 105 Display apparatus, 106 Control apparatus 107 Divided image 1101, 1801, 2201, 2201 Moving object 2101 Panoramic image imaging range

Claims (8)

Photographing means capable of photographing individual photographing areas obtained by dividing the photographing range;
Moving means capable of moving the photographing means to position the angle of view of the photographing means in the different photographing regions;
In an image photographing apparatus comprising the photographing means and a control means for controlling the moving means so as to obtain a photographed image in each photographing region,
The control means detects a moving object that moves in the imaging region based on the acquired captured image, and avoids capturing the moving object in an uncaptured imaging region based on the detection result of the moving object. An image photographing apparatus characterized by controlling photographing means and said moving means. The control means controls the photographing means and the moving means so as to acquire a plurality of photographed images with different photographing conditions in one photographing region,
The control means estimates a subsequent movement in the imaging range by extracting a moving object in the imaging region based on a plurality of captured images with different imaging conditions acquired in one imaging region,
2. The control unit according to claim 1, wherein the control unit controls the photographing unit and the moving unit so as to acquire a photographed image in a photographing region where the moving object does not exist, based on the estimation result of the movement. Image shooting device.   The control means is configured to detect the one unphotographed area before the moving object moves to one unphotographed area or after the moving object passes through the one nonphotographed area based on the estimation result of the movement. The image photographing apparatus according to claim 2, wherein the photographing unit and the moving unit are controlled so as to acquire a photographed image. The control means controls the photographing means and the moving means so as to acquire photographed images in a certain order for the plurality of photographing regions,
The control unit waits until the moving object passes based on the movement estimation result for a shooting area that is determined that the moving object exists based on the movement estimation result, and acquires a captured image. 3. The image photographing apparatus according to claim 2, wherein the photographing means and the moving means are controlled. Photographing means capable of photographing individual photographing areas obtained by dividing the photographing range;
Moving means capable of moving the photographing means to position the angle of view of the photographing means in the different photographing regions;
In an image photographing apparatus comprising the photographing means and a control means for controlling the moving means so as to obtain a photographed image in each photographing region
The control means performs processing for detecting the same moving object in the plurality of acquired captured images, and acquires the captured image again in the imaging area where the same moving object is detected. An image photographing apparatus characterized by controlling the above.   6. The control unit according to claim 1, wherein the control unit outputs a warning when there is an unphotographed area where there is a possibility of photographing the moving object based on the detection result of the moving object. The image capturing device according to item. Comprising display means for displaying the captured images acquired in each imaging region in correspondence with the imaging range;
7. The image photographing apparatus according to claim 1, wherein the control unit performs display based on a detection result of the moving object in association with the photographing range on the display unit. . A plurality of photographing units capable of photographing individual photographing regions obtained by dividing a photographing range and a moving unit capable of positioning the field of view of the photographing unit in different photographing regions by moving the photographing unit. In a program that acquires captured images in the shooting area,
Based on a plurality of photographed images photographed in one photographing region, a moving object in the photographing region is extracted to estimate the subsequent movement in the photographing range. Based on the estimation result of the movement, the moving object is A program for controlling the photographing unit and the moving unit to acquire a photographed image in a non-existing photographing region.

Images