JP6031670B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus such as a digital still camera and a digital video camera. Specifically, the present invention relates to an imaging apparatus using a plurality of imaging optical systems and imaging elements.

  For the purpose of taking a three-dimensional image or the like, an imaging apparatus including a plurality (two) of imaging optical systems and imaging elements is commercially available. In this type of imaging apparatus, in addition to the function of capturing a stereoscopic image, a function for improving the image quality of a moving image or a still image during normal shooting has been proposed and put into practical use. For example, the image capturing apparatus of Patent Document 1 includes at least a still image capturing unit and a moving image capturing unit. The image pickup apparatus selects and records both outputs. As a result, it is possible to obtain a moving image with an appropriate frame rate or a high-definition still image. Patent Document 2 discloses an imaging device that captures a plurality of images with different exposure amounts and combines them. That is, a procedure is disclosed in which a plurality of imaging optical systems are used to sequentially capture a long exposure image and a short exposure image and then combine them. Thus, it is described that a wide dynamic range still image (HDR still image) with parallax is provided.

JP 2003-023555 A JP 2003-018617 A

  However, when the subject moves at high speed as in sports photography or pet photography, there is a problem of subject blurring. This is because the image quality deteriorates due to movement of the subject. For example, when a moving image is shot, an image accumulated during an exposure time of 1/30 second, which is a frame interval, is recorded. Therefore, when the subject moves at a high speed, the image becomes blurred and it is difficult to obtain a clear image. Or, there is a significant problem that the detailed movement of the subject cannot be confirmed during slow playback. Similarly, in the generation of HDR (High Dynamic Range) images, there is a problem that if the position of the subject is different between images having different exposure amounts, the synthesized HDR image becomes unclear.

  Therefore, an object of the present invention is to provide an imaging device that can capture a clear image by preventing image quality deterioration due to subject blurring even when the subject moves at high speed.

The above object is achieved by the following imaging device. That is, an imaging device for photographing a subject,
A first optical system and a second optical system;
A first imaging element that converts an optical image formed by the first optical system into a first electrical signal;
A second imaging element that converts an optical image formed by the second optical system into a second electrical signal;
A first video signal processing unit that reads the first electrical signal from the first image sensor at a first period and generates a first video signal; and the second period from the second image sensor. A second video signal processing unit that reads out at a different second period and generates a second video signal; and a recording unit that records the first video signal and the second video signal on a recording medium at different frame rates. is there.

  According to the present invention, it is possible to provide an imaging apparatus capable of capturing a clear image even when a subject moves at high speed.

External view of imaging apparatus 100 External view of imaging apparatus 100 The block diagram which shows the structure of the imaging device 100 A flowchart for explaining the operation of the imaging apparatus 100 during moving image shooting. A flowchart for explaining the operation of the imaging apparatus 100 during moving image reproduction Flowchart for explaining the operation at the time of high-quality reproduction of the imaging apparatus 100 A flowchart for explaining the operation of the imaging apparatus 100 in the standby state Flowchart for explaining the operation at the time of HDR reproduction of the imaging apparatus 100 Flowchart for explaining the operation at the time of HDR photographing operation of the imaging apparatus 100 Flowchart for explaining AF (autofocus) operation of imaging apparatus 100

[Embodiment 1]
As shown in the external view of FIG. 1, the imaging apparatus 100 includes a first optical system 101 and a second optical system 102. A shutter button 103 is provided in the upper part of the image pickup apparatus 100 so as to determine the timing at the time of still image shooting. In moving image shooting, the shutter button 103 is pressed to start shooting, and the shutter button 103 is pressed again to end shooting.

  A switching button 110 is provided next to the shutter button 103. The switch button 110 is slid horizontally to select ON / OFF of the power supply of the imaging apparatus 100 and an operation mode. That is, it is possible to select any one of power-off, shooting mode, and playback mode.

  As shown in the external view of FIG. 2, the imaging device 100 has a liquid crystal screen 111. The liquid crystal screen 111 functions as a monitor screen in the shooting mode. Further, when the previous switching button 110 is slid to switch to the playback mode, a playback image is displayed. The liquid crystal screen 111 is integrally provided with a touch panel 112 that detects an operation by the user. As a result, the user can select various menus and buttons displayed on the liquid crystal screen 111. In other words, the operation of the imaging apparatus 100 can be controlled via the touch panel 112.

  FIG. 3 is a block diagram illustrating the configuration of the imaging device 100. The imaging apparatus 100 includes a first optical system 101, a second optical system 102, a first imaging element 205, and a second imaging element 206. The first optical system 101 is a zoom optical system. That is, as the first zoom lens 201 moves back and forth, the shooting angle of view changes (magnifies). For example, the angle of view between the wide angle side and the telephoto side can be changed five times. The first optical system 101 also has a first focus lens 203 that adjusts the in-focus position. By moving the first focus lens 203 back and forth, the focus position can be changed from a short distance to infinity.

  The first image sensor 205 converts an optical image formed by the first optical system 101 into a first electric signal 231. The first video signal processing unit 207 reads the first electric signal 231 from the first image sensor 205 at the first period. For example, at the time of moving image shooting, an image is read at a cycle of 1/30 seconds that is a general frame rate. The first video signal processing unit 207 performs noise removal and amplification processing of the first electric signal 231 and outputs a first video signal 241. The first video signal 241 is sent to the video signal recording unit 213 and an image composition unit 222 described later.

  Further, the first video signal processing unit 207 cuts out an arbitrary area in the first video signal 241 and sends it to the first contrast detection unit 209. This transmission timing is also performed for each first period. The first contrast detection unit 209 calculates the contrast value of the received image. Then, the first focus lens 203 of the first optical system 101 is positioned at the position where the calculation result of the contrast value is maximized. In other words, the first focus driving unit 211 drives the first focus lens 203 based on a command from the first contrast detection unit 209. The second optical system 102 has substantially the same configuration as the first optical system 101. The second optical system 102 is a zoom optical system. The second zoom lens 202 moves back and forth to change (magnify) the shooting angle of view. Further, the in-focus position is adjusted by the movement of the second focus lens 204.

  Similarly, the second image sensor 206 converts an optical image formed by the second optical system 102 into a second electric signal 232. The second video signal processing unit 208 reads the second electric signal 232 from the second image sensor 206 at a second period different from the first period. For example, when the first period is 1/30 seconds, reading is performed at a period of 1/300 seconds, which is 10 times faster. The second video signal processing unit 208 performs noise removal and amplification processing on the second electric signal 232 and outputs a second video signal 242. The second video signal 242 is sent to the video signal recording unit 213 and an image composition unit 222 described later.

  The second video signal processing unit 208 cuts out an arbitrary area of the second video signal 242 and sends it to the second contrast detection unit 210. This sending timing is also made to coincide with the second period. The second contrast detection unit 210 calculates the contrast value of the received image every second period. The second contrast detection unit 210 controls the second focus driving unit 212 that drives the second focus lens 204. The second contrast detection unit 210 is also configured to control the first focus driving unit 211.

  The video signal recording unit 213 records the first video signal 241 and the second video signal 242 in the storage medium 220. At this time, the first video signal 241 and the second video signal 242 are recorded on the recording medium 220 at different frame rates. For example, the first video signal 241 is recorded at 30 frames per second, and the second video signal 242 is recorded at 300 frames per second. As the storage medium 220, a semiconductor memory such as an SD card is used. An SD card is a typical example of a portable semiconductor memory. SD card can be replaced with TV or PC. That is, a moving image or a still image recorded by the imaging apparatus 100 can be displayed on a TV or a PC.

  A switching button 110 on the upper part of the imaging apparatus 100 is switched to a reproduction mode. Then, the video signal reproduction unit 221 reads the first video signal 241 and / or the second video signal 242 recorded in the storage medium 220. Then, the image is sent to the image composition unit 222. The image composition unit 222 receives the first video signal 241 and / or the second video signal 242.

  Further, when the switching button 110 is in the shooting mode, it is received from the first video signal processing unit 207 and the second video signal processing unit 208. The image synthesis unit 222 selects and / or synthesizes the first video signal 241 and / or the second video signal 242 and outputs a third video signal 243. For example, the first video signal 241 may be output as it is as the third video signal 243. Alternatively, there may be an image obtained by integrating and synthesizing only the second video signal 242 in the time axis direction. Alternatively, the HDR image may be used as the third video signal 243 by combining the first video signal 241 and the second video signal 242. The contents of the third video signal 243 will be described in detail in the operation of each embodiment.

  The third video signal 243 is sent to the image output unit 223 and displayed on the liquid crystal screen 111 on the back surface of the imaging device 100. That is, the liquid crystal screen 111 functions as a monitor screen in the shooting mode. In addition, when the playback mode is switched, a playback image is displayed.

  Note that an output terminal (not shown) is also connected to the image output unit 223 so that an image can be displayed on an external display device such as a TV or a PC. As an output terminal, an analog output terminal such as an RCA terminal or a digital output terminal such as HDMI standard is generally used.

  The third video signal 243 is also sent to the video signal recording unit 213. The video signal recording unit 213 is configured to record the third video signal 243 on the recording medium 220. The third video signal 243 recorded on the recording medium 220 in this way can also be displayed on the liquid crystal screen 111. That is, the video signal reproduction unit 221 reads the third video signal 243 recorded on the storage medium 220. Then, the image is sent to the image composition unit 222. The image composition unit 222 sends the third video signal 243 to the image output unit 223. In this way, the third video signal 243 is displayed on the liquid crystal screen 111.

  The operation at the time of moving image shooting of the imaging apparatus 100 according to Embodiment 1 configured as described above will be described in order with reference to the flowchart of FIG. In the following, the first period is expressed as a frame rate as N frames per second (N frames per second), and the second period as M frames per second (M frames per second). The second period is referred to as being K times the first period. K = M / N times. This means that the frame rate is K times. For example, the first period is 1/30 seconds and the second period is 1/300 seconds. At this time, N = 30, M = 300, and K = 10.

(Step S100)
The user operates the switching button 110 to select a shooting mode. When the shutter button 103 is pressed, a moving image shooting operation is started. That is, the first image sensor 205 converts the optical image formed by the first optical system 101 into the first electric signal 231. Similarly, the second image sensor 206 converts an optical image formed by the second optical system 102 into a second electric signal 232. The frame number I of the first video signal 241 gives zero as an initial value.

(Step S101)
The frame number I of the first video signal 241 is advanced by 1.

(Step S102)
The first video signal processing unit 207 reads the first electrical signal 231 from the first image sensor 205 at the first period (N frames per second). Then, the first video signal 241 subjected to noise removal and amplification processing is sent to the video signal recording unit 213 and the image synthesis unit 222. At this time, since the first video signal 241 is transmitted as one image, this is referred to as an I-frame image of the first video signal 241.

(Step S103)
The image composition unit 222 selects only the first video signal 241 and outputs it to the image output unit 223 as the third video signal 243. The image output unit 223 displays the third video signal 243, that is, the I-frame image of the first video signal 241 on the liquid crystal screen 111. Thus, the user can take a picture while viewing the monitor image displayed on the liquid crystal screen 111.

(Step S104)
In parallel with step S102, the second video signal processing unit 208 reads the second electrical signal 231 from the second image sensor 206 at the second period (M frames per second). Then, the second video signal 242 subjected to noise removal and amplification processing is sent to the video signal recording unit 213. Since the second video signal 242 is also sent as one image, this is called the J-th frame image of the second video signal 242.

  The second video signal 242 is read out in the second period (M frames per second). Since the second period M is K times the first period N, the frame rate is K times. Therefore, the second video signal 242 is read K times while reading the I frame of the first video signal 241 in step S102. That is, from the J frame of the second video signal 242, J + 1 frame and J + 2 frame are sequentially read out, and images for K frames up to J + K−1 frame are read out.

  Note that the frame number J of the second video signal 242 has a starting value J = K × I because of the relationship K = M / N. Further, the second video signal 242 may not be output to the image composition unit 222. This is because the image composition unit 222 selects only the first video signal 241 as described in step S103.

(Step S105)
The video signal recording unit 213 records the first video signal 241 and the second video signal 242 on the recording medium 220. That is, the image I of the first video signal 241 acquired in step S102 and the K images from the J = K × I frame to the J + K−1 frame of the second video signal 242 acquired in step S104. Record. These may be recorded collectively as the same file. Alternatively, the image of the first video signal 241 and the image of the second video signal 242 may be recorded as separate files.

  Also, index data (index data) indicating the frame position relationship between the first video signal 241 and the second video signal 242 is created. This index data is also recorded in the storage medium 220 by the video signal recording unit 213. That is, it is recorded that the I frame of the first video signal 241 and the J frame to the J + K−1 frame of the second video signal 242 are frames taken at the same time. The index data may be recorded in the same file as the image of the first video signal 241. Alternatively, it may be recorded as an independent separate file.

(Step S106)
When the user presses the shutter button 103 again, the process proceeds to step S107 and the moving image shooting operation is terminated. If the shutter button 103 is not operated, the process returns to step S101 to continue the moving image shooting operation.

(Step S107)
Ends the movie shooting operation.

  Steps S101 to S106 are repeatedly executed in the first cycle. That is, it is repeatedly executed N times per second. As a result, N frame images are acquired from the first video signal 241. On the other hand, an image of K × N frames is acquired as the second video signal 242. Therefore, the frame rate of the second video signal 242 is M = K × N. That is, the second period M is reached.

  The video signal recording unit 213 records the first video signal 241 on the recording medium 220 at the first period (N frames per second). The second video signal 242 is recorded on the recording medium 220 at the second period (M frames per second). Therefore, the first video signal 241 and the second video signal 242 are recorded on the recording medium 220 at different frame rates.

  Note that the multiple K of the first period N and the second period M may not be an integer. For example, when the first period is 1/30 seconds and the second period is 1/100 seconds, N = 30, M = 100, and K = 3.33 may be set.

  In the above moving image shooting operation, the second video signal 242 is continuously recorded on the recording medium 220 (without interruption), but may be intermittently recorded only when designated by the user. That is, when there is no user designation, only the first video signal 241 is recorded. Then, the second video signal 242 is recorded only during the time period designated by the user. For example, consider the case of shooting a 100 m run. The subject will not move at high speed until it stops at the start line. Therefore, only the first video signal 241 is recorded. At the start stage, the user (photographer) designates the start and the recording of the second video signal 242 is started. After the subject exceeds the goal line, the user designates the end and stops the recording of the second video signal 242. During this time, the first video bedding 241 continuously records without interruption. Thus, if the second video signal 242 is intermittently recorded, the recording capacity of the second video signal 242 recorded on the storage medium 220 is reduced. Therefore, there is an advantage that a longer time can be taken.

  At this time (when the second video signal 242 is intermittently recorded), it is better to prepare a ring buffer for a predetermined time in the video signal recording unit 213. That is, the image of the second video signal 242 is always written sequentially into the ring buffer. When the user designates the start, the image is recorded on the recording medium 220 from the image of a predetermined time recorded in the ring buffer. For example, when a pet suddenly starts running due to shooting a pet, the user may hurry to specify the start. Even in such a case, since the image of a predetermined time is stored in the ring buffer, there is an advantage that the second video 242 can be recorded retroactively.

  Further, it may be configured such that motion detection is performed on the first video signal 241 or the second video signal 242, and the recording of the second video signal 242 is automatically performed. That is, the amount of motion is monitored, and the second video signal 242 is recorded only when the amount of motion is intense. When the position of a registered person or pet is detected and the position is being tracked, it is particularly effective to judge based on the amount of movement of the subject. In such a case, providing a ring buffer is more effective.

  Furthermore, the second period may be variable according to the amount of motion. That is, the frame rate of the second video signal 242 may be changed. For example, when the amount of motion is extremely large, K is further increased and K = 20. Conversely, when the amount of motion is not large, K = 5. In this way, the second period (M frames per second) may be variable.

  When the second video signal 242 is intermittently recorded, the state (recording presence / absence) of the second video signal 242 is also recorded in the index data. For example, when only the first video signal 241 is recorded, it is recorded that there is no second video signal 242 corresponding to the frame image of the first video signal 241. On the other hand, when the second video signal 242 is recorded simultaneously, the I-th frame image of the first video signal 241 and the J-th frame to the J + K−1 frame of the second video signal 242 are photographed at the same time. Recorded as a frame that has been recorded.

  Next, the operation at the time of moving image reproduction of the imaging apparatus 100 according to Embodiment 1 will be described in order with reference to the flowchart of FIG.

(Step S200)
The user operates the switching button 110 to select a playback mode. Further, when a playback button is touched from a menu displayed on the liquid crystal screen 111, the touch panel 112 detects the playback button and starts a playback operation. The frame number I of the first video signal 241 gives I = 1 as an initial value.

(Step S201)
The video signal reproduction unit 221 reads the I-th frame image of the first video signal 241 from the recording medium 220. Then, the image of the I frame is sent to the image composition unit 222.

(Step S202)
The image composition unit 222 selects only the first video signal 241 and outputs it to the image output unit 223 as the third video signal 243. The image output unit 223 displays the third video signal 243, that is, the I-frame image of the first video signal 241 on the liquid crystal screen 111.

(Step S203)
The frame number I of the first video signal 241 is advanced by 1.

(Step S204)
When the user touches the stop button from the menu displayed on the liquid crystal screen 111, the process proceeds to S214 and the reproduction operation is terminated. Otherwise, the process proceeds to step S205.

(Step S205)
When the user touches the slow playback button from the menu displayed on the liquid crystal screen 111, the process proceeds to S206, and the slow playback operation is started. If there is no operation by the user, the process returns to step S201 to continue the normal reproduction operation.

  Steps S201 to S205 are repeatedly executed in the first cycle. That is, it is repeatedly executed N times per second. As a result, a moving image photographed by the first optical system 101 can be reproduced at the same speed as during the photographing operation. That is, the user can view a general movie or a moving image similar to a digital camera in real time.

(Step S206)
When the user designates slow reproduction, the recording state of the second video signal 242 is determined. This is because the case where the second video signal 242 is recorded intermittently must be taken into consideration. Therefore, the video signal reproduction unit 221 reads the index data from the recording medium 220. Then, it is determined whether there is a recording result of the second video signal 242 corresponding to the I frame of the first video signal 241 being reproduced. If the corresponding second video signal 242 exists, the process proceeds to step S207. If not, “Slow playback is not possible” is displayed on the liquid crystal screen 111, and the process returns to step S201.

(Step S207)
The frame number J of the second video signal 242 is set with reference to the index data. That is, for the I frame of the first video signal 241, the number of the first frame image of the second video signal 242 captured at the same time is set as J. For example, when the second video signal 242 is continuously recorded on the recording medium 220 (without interruption), J = K × I.

(Step S208)
The video signal reproduction unit 221 reads the J-th frame image of the second video signal 242 from the recording medium 220. Then, the image of the J frame is sent to the image composition unit 222.

(Step S209)
The image composition unit 222 selects only the second video signal 241 and outputs it to the image output unit 223 as the third video signal 243. The image output unit 223 displays the third video signal 243, that is, the image of the J frame of the second video signal 242 on the liquid crystal screen 111.

(Step S210)
The frame number J of the second video signal 242 is advanced by 1.

(Step S211)
When the user touches the stop button from the menu displayed on the liquid crystal screen 111, the process proceeds to S214, and the slow playback operation (and the playback operation) ends. Otherwise, the process proceeds to step S212.

(Step S212)
When the user touches the normal playback button from the menu displayed on the liquid crystal screen 111, the process proceeds to step S213, and the normal playback operation returns. Also, when the J-th frame of the second video signal 242 does not exist, the process proceeds to step S213, and the normal reproduction operation is returned. Otherwise, the process returns to step S208 to continue the slow playback operation.

  Steps S208 to S212 are repeatedly executed in the first cycle. That is, it is repeatedly executed N times per second. Thereby, a moving image shot by the second optical system 102 can be slowly reproduced at a 1 / K times speed during the shooting operation.

(Step S213)
The frame number I of the first video signal 241 is set with reference to the index data. That is, the number of the frame image of the first video signal 241 captured at the same time is set as I for the image of the J frame of the second video signal 242. For example, when the second video signal 242 is continuously recorded on the recording medium 220 (without interruption), the calculation may be performed by I = (J / K). Here, (A) represents the maximum integer that does not exceed the real number A. For example, if A = 10.2, (A) = 10. After setting the frame number I of the first video signal, the process returns to step S201.

(Step S214)
End the slow playback and playback operations.

  As described above, the imaging apparatus 100 according to Embodiment 1 records the first video signal 241 and the second video signal 242 on the recording medium 220 at different frame rates. That is, the video signal recording unit 213 records the first video signal 241 on the recording medium 220 at the first period (N frames per second). The second video signal 242 is recorded on the recording medium 220 at the second period (M frames per second). As a result, during the reproduction operation, the first video signal 241 and the second video signal 242 can be selected and reproduced.

  The second video signal 242 is recorded at a second period (M frames per second) that is K times the first period (N frames per second). Therefore, the exposure time of each frame image of the second video signal 242 is reduced to 1 / K times. Thereby, even when the subject is moving at high speed, there is almost no subject blur in the frame image of the second video signal 242.

  In addition, the second video signal is a moving image having K frames. Therefore, by reproducing the second video signal 242 in the first cycle, slow reproduction at 1 / K times speed can be realized. Moreover, since the number of frames is K times, K times frames per unit time can be displayed as compared with the case where the first video signal 241 is used. Therefore, there is an effect that a fine movement of the subject can be easily confirmed.

  Also, index data indicating the frame positional relationship between the first video signal 241 and the second video signal 242 is created. While the user is playing back the first video signal 241, the user designates a slow playback operation. Then, the frame number of the corresponding second video signal 242 is immediately set with reference to the index data. Therefore, there is an advantage that the slow playback of the second video signal 242 can be started seamlessly.

  In the first embodiment, the second video signal 242 is displayed in the first cycle (N frames per second) during slow playback, but this display cycle may be arbitrarily changed. For example, 2 × N frames may be displayed per second. It may be fixed at 0.5 frames per second.

  Alternatively, in step S210, the frame number may be updated as J = J + 2. That is, it is not necessary to display all the frame images of the second video signal 242 and the thinned images may be displayed.

  In step S206 of the first embodiment, the recording state of the second video signal 242 is determined. Here, when there is no corresponding second video signal 242, the normal playback is continued by returning to step S 201, but slow playback may be performed using the first video signal 241.

[Embodiment 2]
The configuration of imaging apparatus 100 according to the second embodiment is the same as that of the first embodiment shown in FIGS. The moving image shooting operation is also the same as that of the first embodiment (FIG. 4), and the description thereof is omitted.

  The operation at the time of high-quality reproduction of the imaging apparatus 100 according to the second embodiment will be described in order with reference to the flowchart of FIG.

(Step S300)
The user operates the switching button 110 to select a playback mode. Further, when a high-quality playback button is touched from a menu displayed on the liquid crystal screen 111, the touch panel 112 detects the high-quality playback button and starts a high-quality playback operation. The frame number I of the first video signal 241 gives I = 0 as an initial value.

(Step S301)
The frame number I of the first video signal 241 is advanced by 1.

(Step S302)
The recording state of the second video signal 242 is determined. The video signal reproduction unit 221 reads the index data from the recording medium 220. Then, it is determined whether there is a recording result of the second video signal 242 corresponding to the I frame of the first video signal 241. If the corresponding second video signal 242 exists, the process proceeds to step S306, where the integrated image generation and playback operation of the second video signal 242 is performed. If not, the process proceeds to step S303 and the first video signal 241 is reproduced.

(Step S303)
The video signal reproduction unit 221 reads the I-th frame image of the first video signal 241 from the recording medium 220. Then, the image of the I frame is sent to the image composition unit 222.

(Step S304)
The image composition unit 222 selects only the first video signal 241 and outputs it to the image output unit 223 as the third video signal 243. The image output unit 223 displays the third video signal 243, that is, the I-frame image of the first video signal 241 on the liquid crystal screen 111. At this time, it is preferable to display “normal playback” in an overlapping manner on the liquid crystal screen 111.

(Step S305)
When the user touches a stop button or the like from the menu displayed on the liquid crystal screen 111, the process proceeds to S311 to end the high-quality playback operation. If there is no user operation, the process returns to step S301 to continue the reproduction operation.

  The above steps S301 to S305 are repeatedly executed in the first cycle. That is, it is repeatedly executed N times per second.

(Step S306)
The frame number J of the second video signal 242 is set with reference to the index data. That is, for the I frame of the first video signal 241, the number of the first frame image of the second video signal 242 captured at the same time is set as J. For example, when the second video signal 242 is continuously recorded on the recording medium 220 (without interruption), J = K × I.

(Step S307)
The video signal reproduction unit 221 reads K frame images from the J frame to the J + K−1 frame of the second video signal 242 from the recording medium 220. Then, the read K frame images are sent to the image composition unit 222.

(Step S308)
The image synthesizing unit 222 integrates and synthesizes the read K frame images.

(Step S309)
The image composition unit 222 outputs the integrated and synthesized image to the image output unit 223 as an image of the third video signal 243 (the I frame thereof). The image output unit 223 displays the image of the third video signal 243 (I frame thereof) on the liquid crystal screen 111.

(Step S310)
When the user touches a stop button or the like from the menu displayed on the liquid crystal screen 111, the process proceeds to S311 to end the high-quality playback operation. If there is no user operation, the process returns to step S301 to continue the reproduction operation.

  The processes in steps S301 to S310 are also repeatedly executed in the first cycle. That is, it is repeatedly executed N times per second.

(Step S311)
Ends the high-quality playback operation.

  In this way, the integrated image of the second video signal 242 is displayed as the third video signal 243. The second video signal is recorded at a second period (M frames per second) K times the first period (N frames per second). The exposure time of each frame image of the second video signal 242 is reduced to 1 / K times. Therefore, even when the subject is moving at high speed, there is almost no subject blur in the frame image of the second video signal 242. Therefore, in step S308, motion detection between frames of the second video signal 242 is performed, and integration according to the motion amount is performed. An area where the movement of each frame image is small may be added as it is. A portion where the subject moves largely may be added while correcting the subject position in consideration of the amount of movement. Alternatively, weighted addition may be performed according to the amount of motion. In this way, it is possible to synthesize a clear image with less subject blur compared to the first video signal 241.

  Alternatively, in step S308, an image to be integrated can be selected according to the motion detection result. For example, among the K frames from the J-th frame of the second video signal 242, if the motion is small in the preceding K1 frame and the motion is large in the subsequent K2 frames, the preceding K1 frames May be accumulated. If the motion is small in all frames, all K frames may be integrated. When it is determined that the motion is large in all frames, an image with the least noise may be selected and the third video signal 243 may be used without performing the integration operation.

  Further, the integrated image has a feature that random noise can be eliminated. Each frame image includes randomly generated noise. However, since the generation position is random, this can be canceled by integrating and combining a plurality of images. It is also possible to detect pixels that appear to be noise from image comparison between frames and interpolate using luminance and color information of surrounding pixels. In this way, static noise can be reduced.

  Further, the above-described various integration methods may be performed while sequentially selecting the method that provides the highest image quality. Alternatively, an image with the highest image quality (that is, no subject blurring and less noise) may be selected after a plurality of candidate images are created by performing all the integration methods.

  As described above, the various integration methods can be selected in step S308 because the integration process is performed during reproduction. In general, it is known that in a semiconductor memory such as an SD card used as the recording medium 220, the recording speed and the reproduction speed are greatly different. That is, the writing speed (recording speed) for writing a certain amount of digital data is about 2 to 10 times slower than the reading speed (playback speed). When the second video signal 242 is recorded in the second period (M frames per second) at a high frame rate, much time is required only by recording. However, in the second embodiment, since the second video signal 242 is integrated during reproduction, it can be read from the recording medium 220 at high speed. Therefore, there is an advantage that a sufficient processing time can be secured.

  Further, the playback frame rate of the third video signal 243 is made to coincide with the same first period (N frames per second) as that of the first video signal 241, so that it is not necessary to limit the viewing target to the liquid crystal screen 111. Usually, a frame rate of 30 frames per second or the like is regularly used as the first period. At this time, it is only necessary to connect the output from the image output unit 223 to an external device such as a TV or a PC. Then, not only the first video signal 241 but also the third video signal 243 can be viewed. That is, there is an advantage that a clear image with small subject blur can be viewed on a wide variety of devices.

  Next, the operation in the standby state of the imaging apparatus 100 according to the second embodiment will be described in order with reference to the flowchart of FIG.

(Step S400)
If the user operates the switching button 110 to select a playback mode or a shooting mode and then continues the standby state for a long time, the imaging apparatus 100 starts a standby high-quality recording operation. The frame number I of the first video signal 241 gives I = 0 as an initial value.

(Step S401)
The frame number I of the first video signal 241 is advanced by 1.

(Step S402)
The recording state of the second video signal 242 is determined. The video signal reproduction unit 221 reads the index data from the recording medium 220. Then, it is determined whether there is a recording result of the second video signal 242 corresponding to the I frame of the first video signal 241. If the corresponding second video signal 242 exists, the process proceeds to step S404, and the integrated image generation and recording operation of the second video signal 242 is performed. If not, the process proceeds to step S403.

(Step S403)
When the user operates the shutter button 103 or the switching button 110 or operates the touch panel 112, the user is no longer in a standby state. In step S409, the standby high image quality recording operation is stopped. If there is no user operation, the process returns to step S401.

(Step S404)
The frame number J of the second video signal 242 is set with reference to the index data. That is, for the I frame of the first video signal 241, the number of the first frame image of the second video signal 242 captured at the same time is set as J. For example, when the second video signal 242 is continuously recorded on the recording medium 220 (without interruption), J = K × I.

(Step S405)
The video signal reproduction unit 221 reads K frame images from the J frame to the J + K−1 frame of the second video signal 242 from the recording medium 220. Then, the read K frame images are sent to the image composition unit 222.

(Step S406)
The image synthesizing unit 222 integrates and synthesizes the read K frame images. The integration method is preferably the same process as in step S308.

(Step S407)
The image synthesis unit 222 outputs the integrated and synthesized image to the video signal recording unit 213 as an image of the third video signal 243. The video signal recording unit 213 records the received third video signal 243 on the recording medium 220.

  Also, index data (index data) indicating the frame position relationship between the first video signal 241 and the third video signal 243 may be created and recorded in the storage medium 220. That is, it is recorded that the I frame of the first video signal 241 and the L frame of the third video signal 243 are frames taken at the same time.

(Step S408)
When the user operates the shutter button 103 or the switching button 110 or operates the touch panel 112, the user is no longer in a standby state. In step S409, the standby high image quality recording operation is stopped. If there is no user operation, the process returns to step S401.

  What is necessary is just to implement the process of the above steps S401-S408 taking arbitrary time. That is, there is no restriction such as “repeatedly executed N times per second”.

(Step S409)
Ends the standby high-quality recording operation.

  In this way, the accumulated image of the second video signal 242 is synthesized at the time of standby, and is recorded on the recording medium 220 as the third video signal 243. Since there is more time in standby than during playback, it is possible to record a higher quality image (no subject blurring and clear).

  At the time of reproduction, the presence of the recorded third video signal 243 is first checked with reference to the index data, and if present, the video signal reproduction unit 221 reads it. The image composition unit 222 selects the read third video signal 243 and outputs it to the image output unit 223.

  In the second embodiment, one integrated image is synthesized from the K frame images. However, a plurality of accumulated images can be synthesized. For example, integration may be performed while shifting three frames out of K frames. That is, an image obtained by integrating J, J + 1, and J + 2 frames, an image obtained by integrating J + 1, J + 2, and J + 3 frames, and an image obtained by integrating J + K-3, J + K-2, and J + K-1 frames may be sequentially combined. In this way, K-2 integrated images are synthesized from the K images. In this way, it is possible to synthesize a plurality of clear images with no subject blurring and eliminating random noise. The K-2 (multiple) accumulated images are recorded on the recording medium 220 as the third video signal 243. Then, the slow reproduction similar to the first embodiment is also possible. That is, when the slow reproduction is selected, the third video signal 243 may be handled as the second video signal 242 of the first embodiment. Slow playback using clearer and less noise images can be realized.

  In the second embodiment, the reproduction operation after moving image shooting has been described. However, the present invention is not limited to this. It can also be applied to still image shooting and continuous shooting. Each frame image may be regarded as one still image.

[Embodiment 3]
Since the configuration of imaging apparatus 100 according to Embodiment 3 is the same as that of Embodiment 1 shown in FIGS. 1, 2, and 3, description thereof will be omitted. The moving image shooting operation is also the same as that of the first embodiment (FIG. 4), and the description thereof is omitted.

  The operation at the time of HDR reproduction of the imaging apparatus 100 according to the third embodiment will be described in order with reference to the flowchart of FIG.

(Step S500)
The user operates the switching button 110 to select a playback mode. Further, when an HDR playback button is touched from a menu displayed on the liquid crystal screen 111, the touch panel 112 detects the HDR playback button and starts an HDR playback operation. The frame number I of the first video signal 241 gives I = 0 as an initial value.

(Step S501)
The frame number I of the first video signal 241 is advanced by 1.

(Step S502)
The video signal reproduction unit 221 reads the I-th frame image of the first video signal 241 from the recording medium 220. Then, the image of the I frame is sent to the image composition unit 222.

(Step S503)
The recording state of the second video signal 242 is determined. The video signal reproduction unit 221 reads the index data from the recording medium 220. Then, it is determined whether there is a recording result of the second video signal 242 corresponding to the I frame of the first video signal 241. If the corresponding second video signal 242 exists, the process advances to step S506 to perform HDR image generation and playback operation. If not, the process proceeds to step S504, where the first video signal 241 is reproduced.

(Step S504)
The image composition unit 222 selects only the first video signal 241 and outputs it to the image output unit 223 as the third video signal 243. The image output unit 223 displays the third video signal 243, that is, the I-frame image of the first video signal 241 on the liquid crystal screen 111.

(Step S505)
When the user touches a stop button or the like from the menu displayed on the liquid crystal screen 111, the process proceeds to step S512, and the reproduction operation ends. If there is no user operation, the process returns to step S501 to continue the reproduction operation.

  The above steps S501 to S505 are repeatedly executed in the first cycle. That is, it is repeatedly executed N times per second.

(Step S506)
The frame number J of the second video signal 242 is set with reference to the index data. That is, for the I frame of the first video signal 241, the number of the first frame image of the second video signal 242 captured at the same time is set as J. For example, when the second video signal 242 is continuously recorded on the recording medium 220 (without interruption), J = K × I.

(Step S507)
The video signal reproduction unit 221 reads K frame images from the J frame to the J + K−1 frame of the second video signal 242 from the recording medium 220. Then, the read K frame images are sent to the image composition unit 222.

(Step S508)
The image synthesizing unit 222 integrates and synthesizes the read K frame images. The integration method is preferably the same processing as step S308 in the second embodiment.

(Step S509)
The image synthesis unit 222 synthesizes an HDR image from an image obtained by integrating and synthesizing the second video signal 242 and an I-frame image of the first video signal 241.

(Step S510)
The synthesized HDR image is output to the image output unit 223 as the third video signal 243 (the I frame). The image output unit 223 displays the image of the third video signal 243 (I frame thereof) on the liquid crystal screen 111.

(Step S511)
When the user touches a stop button or the like from the menu displayed on the liquid crystal screen 111, the process proceeds to step S512, and the HDR reproduction operation is terminated. If there is no user operation, the process returns to step S501 to continue the HDR reproduction operation.

  The processes in steps S501 to S511 are also repeatedly executed in the first cycle. That is, it is repeatedly executed N times per second.

(Step S512)
Ends the high-quality playback operation.

  The synthesis of the HDR image (wide dynamic range image) in step S509 is performed as follows, for example. The image of the I frame of the first video signal 241 is a long exposure image. An image obtained by integrating and synthesizing K images from the J frame of the second video signal 242 is a short-time exposure image. Both images are separated into a luminance signal and a color difference signal, and an appropriate exposure area of each image is obtained. Next, gradation correction is performed on each appropriate exposure area. Then, both images are combined using only the appropriate exposure area. In this way, it is possible to synthesize an image with a wide dynamic range and an HDR image by combining the appropriate exposure region of the long-time exposure image and the appropriate exposure region of the short-time exposure image.

  Since the first video signal 241 and the second video signal 242 are images taken at the same time, an HDR image without subject blur can be synthesized even when the subject moves at high speed. Moreover, it can be continuously displayed as an HDR video.

  In general, a noise component tends to remain in an image having a short exposure time, and random noise is particularly noticeable in many cases. However, in the third embodiment, since the integrated composite image of the second video signal 242 is used as the short-time exposure image, noise can be eliminated at the time of stacking synthesis. Thereby, the noise of the HDR composite image can be reduced.

  Noise also has an adverse effect on the HDR image composition process. In an image with a significant noise component, the proper exposure range is often wrong. In addition, there is a case where tone correction is failed due to the brightness of noise. However, since the noise of the short-time exposure image is small in the third embodiment, the appropriate exposure range and gradation correction can be performed with high accuracy, and a clearer HDR composite image can be obtained.

  As in the second embodiment, the HDR image composition process may be performed during standby. In this case, the image composition unit 222 generates an HDR image and outputs the HDR image to the video signal recording unit 213 as the third video signal 243. The video signal recording unit 213 records the received third video signal 243 on the recording medium 220.

  Furthermore, as described in the second embodiment, for example, integration may be performed while shifting by three frames. As a result, since K-2 integrated composite images can be generated, they may be individually combined with the I-frame image of the first video signal 241. Then, the slow reproduction similar to the first embodiment is also possible. That is, slow playback of HDR images becomes possible.

  In the third embodiment, the playback operation after moving image shooting has been described, but the present invention is not limited to this. It can also be applied to still image shooting and continuous shooting. Each frame image may be regarded as one still image.

[Embodiment 4]
Since the configuration of imaging apparatus 100 according to Embodiment 4 is the same as that of Embodiment 1 shown in FIGS. 1, 2, and 3, description thereof is omitted.

  The operation at the time of the HDR imaging operation of the imaging apparatus 100 according to Embodiment 4 will be described in order with reference to the flowchart of FIG.

(Step S600)
The user operates the switching button 110 to select a shooting mode. Further, when the HDR shooting button is touched from the menu displayed on the liquid crystal screen 111, the mode is switched to the HDR shooting mode. When the shutter button 103 is pressed, the HDR shooting operation is started. That is, the first image sensor 205 converts the optical image formed by the first optical system 101 into the first electric signal 231. Similarly, the optical image formed by the second optical system 102 is converted into a second electrical signal 232 by the second imaging element 206. The frame number I of the first video signal 241 gives zero as an initial value.

(Step S601)
The frame number I of the first video signal 241 is advanced by 1.

(Step S602)
The first video signal processing unit 207 reads the first electrical signal 231 from the first image sensor 205 at the first period (N frames per second). Then, noise removal and amplification processing are performed, and the I-frame image of the first video signal 241 is sent to the image composition unit 222.

(Step S603)
In parallel with step S602, the second video signal processing unit 208 reads the second electric signal 231 from the second image sensor 206 at a second period (M frames per second). Then, noise removal and amplification processing are performed, and images for K frames from the J frame to the J + K−1 frame of the second video signal 242 are sent to the image composition unit 222. Note that J = K × I.

(Step S604)
The image synthesizing unit 222 integrates and synthesizes the received K frame images. The integration method is preferably the same processing as step S308 in the second embodiment.

(Step S605)
The image synthesis unit 222 synthesizes an HDR image from an image obtained by integrating and synthesizing K frame images of the second video signal 242 and an I-frame image of the first video signal 241. Then, the synthesized HDR image is output to the image output unit 223 and the video signal recording unit 213 as the third video signal 243 (the I frame). The HDR image synthesizing method may be the same as that in step S509 in the third embodiment.

(Step S606)
The image output unit 223 displays the received image of the third video signal 243 (I frame thereof) on the liquid crystal screen 111.

(Step S607)
The video signal recording unit 213 records the received third video signal 243 on the recording medium 220. Also, index data (index data) indicating the frame position relationship between the first video signal 241 and the third video signal 243 may be created and recorded in the storage medium 220. That is, it is recorded that the I frame of the first video signal 241 and the L frame of the third video signal 243 are frames taken at the same time.

(Step S608)
If the user presses the shutter button 103 again, the process advances to step S609 to end the HDR shooting operation. If the shutter button 103 is not operated, the process returns to step S601 to continue the HDR shooting operation.

(Step S609)
The HDR shooting operation is terminated.

  As described above, in the fourth embodiment, the combined image synthesis and the HDR image synthesis processing are performed at the time of shooting, and the result is displayed on the liquid crystal screen 111. Therefore, there is an advantage that the user can continue shooting while confirming the HDR image composition result.

  Further, the synthesized HDR image is recorded on the recording medium 220 as the third video signal 243. The third video signal 243 is recorded at the same frame rate (N frames per second) as the first video signal 241. Therefore, HDR recording can be performed for a long time without reducing the capacity of the storage medium 220.

  Note that step S605 may be skipped (not performed), and a high-quality moving image based on the integrated image of the second video signal 242 may be displayed and recorded as the third video signal 243.

  In the fourth embodiment, the operation at the time of moving image shooting has been described. However, the present invention is not limited to this. It can also be applied to still image shooting and continuous shooting. Each frame image may be regarded as one still image.

[Embodiment 5]
Since the configuration of imaging apparatus 100 according to Embodiment 5 is the same as that of Embodiment 1 shown in FIGS. 1, 2, and 3, description thereof will be omitted.

  The AF (autofocus) operation of the imaging apparatus 100 according to the fifth embodiment will be described in order with reference to the flowchart of FIG.

(Step S700)
When the user operates the switching button 110 to select a shooting mode, an AF operation is started.

(Step S701)
The second image sensor 206 converts an optical image formed by the second optical system 102 into a second electric signal 232. The second video signal processing unit 208 reads out the second electric signal 232 from the second image sensor 206 at the second period.

(Step S702)
The second video signal processing unit 208 cuts out an arbitrary area of the second video signal 242 and sends it to the second contrast detection unit 210. The second contrast detection unit 210 calculates the contrast value of the received image.

(Step S703)
The in-focus state of the second optical system 102 is determined from the contrast value calculated by the second contrast detection unit 210 or the like. If it is determined that the subject is in focus, the process proceeds to step S705. If the in-focus state is not confirmed, the process proceeds to step S704.

(Step S704)
The second focus driving unit 212 is controlled to move the second focus lens 204. Then, the process returns to step S701.

  As described above, the processes in steps S701 to S704 are repeatedly executed in the second period. That is, it is repeatedly executed M times per second. Thus, the focus search operation is executed. As an algorithm for the focus search operation, a hill-climbing method that sequentially compares contrast values by using wobbling (vibration with a minute amplitude) is preferable. Alternatively, a method of predicting a peak position of a contrast value after searching a certain section to obtain a plurality of contrast values may be used.

(Step S705)
When the focus search operation is completed, the second focus driving unit 212 is controlled to move the second focus lens 204 to the focus position. At the same time, the first focus driving unit 211 is controlled to move the first focus lens 203 to the in-focus position.

(Step S706)
When the user operates the switching button 110 to select the power OFF or playback mode, the process proceeds to step S707 and the AF operation is terminated. If there is no user operation, the process returns to step S701 to continue the AF operation.

(Step S707)
The AF operation ends.

  As described above, in the fifth embodiment, since the AF operation of the second optical system 102 is performed in the second period (M frames per second), it is possible to instantly cope with a change in the in-focus position due to high-speed movement of the subject. . Moreover, the first optical system 101 that captures images in the first period (N frames per second) also has an effect of eliminating the focus shift caused by the high-speed movement of the subject. In other words, the AF operation of the second optical system 102 is performed in the second cycle that is K times faster during the exposure of the I-frame image of the first video signal 241. Therefore, before the exposure of the first frame of the first video signal 241 is completed, the AF search operation is completed, the first focus driving unit 211 is controlled, and the first focus lens 203 is also connected to the first optical system 101. Move to the focal position.

  Note that the relationship between the first period and the second period is not necessarily M> N, and may be M <N. For example, when a part of the background is very bright due to a strong light source such as the sun, a subject (for example, a face) facing away from the light source may become dark. In such a case, when it is desired to focus on the face of the subject, M <N and the second optical system may be exposed for a long time. Although the AF operation speed is slow, the exposure time can be lengthened, so that even a dark face can be accurately focused.

  The first contrast detection unit 209 may perform a normal operation. The first focus driving unit 211 follows a command from the first contrast detection unit 209. Then, only when there is a command from the second contrast detection unit 210, this should be prioritized.

  The present invention is suitable for an imaging apparatus such as a digital still camera and a digital video camera.

DESCRIPTION OF SYMBOLS 100 Imaging device 101 1st optical system 102 2nd optical system 103 Shutter button 110 Switching button 111 Liquid crystal screen 112 Touch panel 201 1st zoom lens 202 2nd zoom lens 203 1st focus lens 204 2nd focus lens 205 1st image sensor 206 Second imaging element 207 First video signal processing unit 208 Second video signal processing unit 209 First contrast detection unit 210 Second contrast detection unit 211 First focus driving unit 212 Second focus driving unit 213 Video signal recording unit 220 Recording medium 221 Video signal reproduction unit 221
222 Image composition unit 223 Image output unit 231 First electric signal 232 Second electric signal 241 First video signal 242 Second video signal 243 Third video signal

Claims (8)

An imaging device for photographing a subject,
A first optical system and a second optical system;
A first imaging element that converts an optical image formed by the first optical system into a first electrical signal;
A second imaging element that converts an optical image formed by the second optical system into a second electrical signal;
A first video signal processor that reads out the first electrical signal from the first image sensor at a first period and generates a first video signal;
A second video signal processing unit that reads the second electrical signal from the second imaging element in a second cycle shorter than the first cycle, unlike the first cycle, and generates a second video signal;
A recording unit for recording the first video signal and the second video signal on a recording medium at different frame rates;
A reproducing unit for reading out the first video signal and / or the second video signal from the recording medium;
An input receiving unit that receives input from the user,
The reproduction unit reproduces the first video signal when the input reception unit receives a normal reproduction instruction, and reproduces the second video signal when the input reception unit receives a slow reproduction instruction;
Imaging device. When the playback unit is playing back the first video signal, and the input receiving unit receives a slow playback instruction, the location corresponding to the first video signal played back when the slow playback command is received To play the second video signal from
The imaging device according to claim 1. The reproduction unit reproduces the first video signal when the input reception unit accepts a slow reproduction instruction when the first video signal is being reproduced, and the second video signal does not exist;
The imaging device according to claim 2. The recording unit creates index data indicating a frame position relationship between the first video signal and the second video signal, and the playback unit refers to the index data when the input reception unit receives the slow playback instruction. To determine the playback position of the second video signal,
The imaging device according to claim 2. An imaging device for photographing a subject,
A first optical system and a second optical system;
A first imaging element that converts an optical image formed by the first optical system into a first electrical signal;
A second imaging element that converts an optical image formed by the second optical system into a second electrical signal;
A first video signal processor that reads out the first electrical signal from the first image sensor at a first period and generates a first video signal;
A second video signal processing unit that reads the second electrical signal from the second imaging element in a second cycle shorter than the first cycle, unlike the first cycle, and generates a second video signal;
A recording unit for recording the first video signal and the second video signal on a recording medium at different frame rates;
A reproduction unit for reading out the first video signal and / or the second video signal from the recording medium,
The recording unit performs motion detection on the first video signal, and records the second video signal only when a result of the motion detection is a predetermined value or more.
Imaging device. An imaging device for photographing a subject,
A first optical system and a second optical system;
A first imaging element that converts an optical image formed by the first optical system into a first electrical signal;
A second imaging element that converts an optical image formed by the second optical system into a second electrical signal;
A first video signal processor that reads out the first electrical signal from the first image sensor at a first period and generates a first video signal;
A second video signal processing unit that reads the second electrical signal from the second imaging element in a second cycle shorter than the first cycle, unlike the first cycle, and generates a second video signal;
A recording unit for recording the first video signal and the second video signal on a recording medium at different frame rates;
An image synthesis unit for generating a third video signal obtained by integrating and synthesizing the first video signal and / or the second video signal;
A playback unit that reads the first video signal and / or the second video signal and / or the third video signal from the recording medium,
Imaging device. The combining unit generates a fourth image signal obtained by synthesizing the third image signal and the first video signal Toka et HDR image,
The reproducing unit reads the first video signal and / or the second video signal and / or the third video signal and / or the fourth video signal from the recording medium;
The imaging device according to claim 6. The recording unit records the output of the image composition unit on the recording medium;
The imaging device according to claim 6.

Images