JP2010219693A - Imaging device and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent an influence on a subject detection resulting from a change of a phase of an imaging signal generated by a change of a drive method of an image pickup device. <P>SOLUTION: An imaging device includes: an image pickup device drive portion 4 controlling a drive system of an image pickup device 3 composed of a plurality of pixels; a subject detecting portion 7 recognizing a predetermined subject by an image signal outputted from the image pickup device 3 and extracting first subject information; a subject detection frame display control portion 9 processing the first subject information and generating second subject information superimposed on an input image to a display device 14; and a phase control portion 6 controlling a phase of an input signal for the subject detecting portion 7 based on the drive system of the image pickup device 3. The imaging device prevents an influence on a subject detection resulting from a change of a phase of an image signal generated by a change of the drive system of the image pickup device 3. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method, and more particularly, to a technique suitable for use so that a change in the phase of an image signal caused by changing a driving method of an imaging element does not affect subject detection.

  In recent years, in a digital camera device, a technique for estimating the position and size of a specific subject such as a human face using an image recognition algorithm has been put into practical use. Information on the position and size of a specific subject is used, for example, for autofocus control.

  That is, by setting a distance measurement frame in accordance with the above-described subject position and size, generating autofocus evaluation data, and performing autofocus control, focus control focusing on a specific subject can be performed. At the same time, what the specific subject is like is also displayed to the user of the digital camera device.

  Conventionally, as in Patent Document 1, a subject is detected from continuously acquired images, and at the same time, information (for example, a frame) indicating a specific subject is displayed on a display image on an LCD display unit mounted on the digital camera device body. There is known a method for displaying the images in a superimposed manner.

Japanese Patent Laid-Open No. 2007-30641

  However, when the subject is detected and the frame is displayed by the above-described method, the detected position of the detected subject is detected when the phase of the image signal is changed by changing the driving method of the image sensor. There is a problem that the displayed frame is not stable.

  For example, as shown in FIG. 5, the combination of addition in the even field (5-2) and the odd field (5-3) is changed with respect to the frame image (5-1) which is all pixels at the time of reading from the image sensor. There is an imaging method for reading out as an interlaced image. In this imaging method, since the phase difference of the image signal is generated for each field, the result of subject detection varies even if the subject is stationary, resulting in a phenomenon that the frame vibrates finely.

  The present invention has been made in view of the above-described problems, and it is an object of the present invention to prevent a change in the phase of an image signal caused by changing the driving method of an image sensor from affecting the object detection.

  An image pickup apparatus according to the present invention recognizes a predetermined subject from an image pickup device including a plurality of pixels, a drive control unit that controls a drive method of the image pickup device, and an image signal output from the image pickup device. Subject detection means for extracting information; subject information display control means for processing the first subject information to generate second subject information and superimposing it on an input image to the image display means; and driving of the image sensor And a phase control means for controlling the phase of the input signal to the subject detection means based on the method, so that the change in the phase of the image signal caused by changing the driving method of the image sensor does not affect the subject detection. It is characterized by that.

  According to the present invention, since the phase control for controlling the phase of the input signal to the subject detection means is performed based on the drive method, the change in the phase of the image signal caused by the change in the drive method of the image sensor is detected. It is possible not to affect the subject detection. Thereby, subject detection and display can be stabilized.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. It is a timing chart which shows the 1st Embodiment of this invention and demonstrates operation | movement of a phase control part. It is a block diagram which shows the 2nd Embodiment of this invention and illustrates schematic structure of an imaging device. It is a timing chart which shows the 2nd Embodiment of this invention and demonstrates operation | movement of a phase control part. It is a figure explaining the even field image and odd field image which comprise the frame image read from an image sensor. 5 is a flowchart illustrating a procedure for detecting a subject and displaying a frame according to the first embodiment of this invention. 10 is a flowchart illustrating a procedure for detecting a subject and displaying a frame according to the second embodiment of this invention.

(First embodiment)
FIG. 1 shows a first embodiment of the present invention, which is an example applied to a video camera device (hereinafter referred to as an imaging device) 100.
In FIG. 1, 1 is a lens, 2 is a lens drive unit, 3 is an image sensor, 4 is an image sensor drive unit, 5 is an analog front end unit (shown as AFE) including a CDS circuit and an A / D converter, 6 Is a phase control unit.

  Reference numeral 7 denotes a subject detection unit, 8 denotes a camera signal processing unit, and 9 denotes a subject detection frame display control unit that performs subject information display control. Reference numeral 10 denotes an image superimposing unit that superimposes the frame signal from the subject detection frame display control unit 9 on the image signal from the camera signal processing unit 8, and 12 denotes a recording medium that can be detached from the imaging apparatus. Reference numeral 11 denotes a recording processing unit that records an image signal subjected to interframe predictive coding processing on the recording medium 12, and reference numeral 14 denotes a display device that displays the image signal subjected to camera signal processing. Reference numeral 13 denotes a display processing unit that displays an image on the display device 14, 15 denotes a bus, and 16 denotes a system control unit that controls the overall operation of the imaging device 100.

  Hereinafter, a photographing operation performed by the imaging apparatus 100 configured as described above will be described. First, the lens 1 is driven by a driving pulse from the lens driving unit 2 controlled by the system control unit 16 to form a subject image on the image sensor 3.

  The image pickup device 3 has a plurality of pixels, picks up an image with a drive pulse sent from the image pickup device drive unit 4 controlled by the system control unit 16, and generates and outputs an analog image signal. The analog image signal output from the image pickup device 3 is subjected to clock synchronization noise removal by a CDS circuit inside the AFE 5 and converted into a digital image signal by an A / D converter provided inside the AFE 5.

  The digital image signal output from the AFE 5 is converted into an image signal of luminance color difference by the camera signal processing unit 8. The phase control unit 6 determines whether to perform subject detection according to the driving state of the image sensor 3 and supplies detection control information to the subject detection unit 7.

  The subject detection unit 7 detects the subject from the image signal according to the detection control information supplied from the phase control unit 6, generates subject detection information, and outputs it to the system control unit 16 via the bus 15. The system control unit 16 outputs the subject detection information output from the subject detection unit 7 to the subject detection frame display control unit 9. The subject detection frame display control unit 9 processes the first subject information input from the bus 15, calculates frame information (frame signal) to be superimposed on the display image as second subject information, and calculates the calculated frame. Information is output to the image superimposing unit 10. The image superimposing unit 10 superimposes the frame information input from the subject detection frame display control unit 9 on the input image signal output to the display device 14 that performs image display.

  The subject detection unit 7 extracts the position / size information of the subject as the first subject information. Then, subject detection is performed by image recognition from the extracted subject position / size information and the like. The subject detection unit 7 of the present embodiment detects a human face as an example of a subject from the image signal. Then, the system control unit 16 is notified of information (face information) related to the face area, such as the position and range (size) of the detected face, and reliability (probability of being a face). The face position may be the center coordinates of the face area.

  A known face detection technique can be used for subject detection in this embodiment. As a known face detection technique, a method based on learning using a neural network or the like, template matching is used to search a part having a characteristic shape of eyes, nose, mouth, etc. from an image, and if the degree of similarity is high, it is regarded as a face There are methods. In addition, many other methods have been proposed, such as a method that detects image feature amounts such as skin color and eye shape and uses statistical analysis. In general, a plurality of these methods are combined to improve face detection accuracy. Specific examples include a face detection method using wavelet transform and image feature amount described in JP-A-2002-251380.

  The image superimposing unit 10 superimposes a frame based on the frame information output from the subject detection frame display control unit 9 on the image signal. The display processing unit 13 converts the image signal on which the frame output from the image superimposing unit 10 is superimposed into a display signal suitable for the display device 14 (for example, an NTSC analog signal) and displays the display signal on the display device 14. .

  The recording processing unit 11 compresses and encodes the image signal output from the camera signal processing unit 8, converts it into data suitable for the recording medium 12 (for example, file system data having a hierarchical structure), and records the data on the recording medium 12. . The system control unit 16 performs drive control of the lens driving unit 2. Further, it controls the operations of the image sensor driving unit 4, the camera signal processing unit 8, the recording processing unit 11, and the display processing unit 13.

FIG. 2 is a timing chart showing the operation of the phase controller 6.
In FIG. 2, 2-1 represents a synchronization signal in units of fields. 2-2 represents an output signal from the image pickup device 3, and the even field is E, the odd field is O, and the numbers are serial numbers. It is assumed that the even field and the odd field are different in the driving method of the image sensor 3, and images having different phases are output.

  Reference numeral 2-3 denotes control information generated by the phase controller 6 so as to become active in the same phase field based on the imaging method. In this embodiment, the operation is performed so that the output signal of the even field becomes active.

  Reference numeral 2-4 denotes subject information detected by the subject detection unit 7. The subject detection unit 7 performs a detection operation in a field in which the control information output from the phase control unit 6 is active, outputs a detection result after one field, and maintains the previous output result when the control information is not active. Shall. Therefore, the detection result DET_E0 of the E0 field is output. In this embodiment, the control information is active in all even fields, but it is not necessary to be active in all fields if the phase is the same.

Next, the processing procedure of this embodiment will be described with reference to the flowchart of FIG.
When the processing is started, shooting by the image sensor 3 is started in step S61, and an analog image signal is generated and output.

In step S62, the clock synchronization noise is removed from the analog image signal output from the image sensor 3, and a digital image signal is generated by A / D conversion.
Next, proceeding to step S63, the digital image signal generated at step S62 is converted into an image signal of luminance color difference.

  Next, in step S64, it is determined whether to perform subject detection according to the driving state of the image sensor 3. If subject detection is performed, the process proceeds to step S65. If subject detection is not performed, the process proceeds to step S66.

  In step S65, in accordance with the detection control information supplied from the phase control unit 6, the subject is detected from the image signal, subject detection information is generated and output, and held in the memory. Thereafter, the process proceeds to step S67. On the other hand, in step S66, subject detection information is not generated, and the subject detection information held in the memory is output to the system control unit 16 via the bus 15. Thereafter, the process proceeds to step S67.

  In step S67, frame information to be superimposed on the display image is calculated from the subject detection information output from step S65 or step S66, and the calculated frame information is output to the image superimposing unit 10.

  Next, in step S68, the frame is superimposed on the image signal based on the frame information generated in step S67. Next, in step S69, the image signal with the frame superimposed thereon is converted into a display signal suitable for the display device 14 (for example, an NTSC analog signal or the like) and displayed on the display device 14. Next, in step S610, it is determined whether all the processes have been completed. If the result of this determination is not complete, processing returns to step S61 and the above-described processing is performed. If the result of determination in step S610 is that the processing has ended, end processing is performed.

  In the present embodiment, when the processing as described above is performed, the driving method of the image sensor 3 is changed between the even field and the odd field, so that images having different phases are output. There is. Even in such a case, it is possible not to affect the subject detection. Thereby, subject detection and display can be stabilized. The change of the driving method is, for example, changing the combination of pixel addition or thinning periodically so that the output signal becomes an interlace signal.

(Second Embodiment)
FIG. 3 shows a second embodiment of the present invention and is a block diagram illustrating an example in which the present invention is applied to a video camera device as an imaging device.
In FIG. 3, 1 is a lens, 2 is a lens driving unit, 3 is an image sensor, 4 is an image sensor driving unit, 5 is an analog front end unit (shown as AFE) including a CDS circuit and an A / D converter, 6 Is a phase control unit, 7 is a subject detection unit, and 8 is a camera signal processing unit.

  9 is a subject detection frame display control unit, 10 is an image superimposing unit that superimposes frame information from the subject detection frame display control unit 9 on an image signal from the camera signal processing unit 8, and 12 is a recording medium that is removable from the imaging apparatus. It is.

  Reference numeral 11 denotes a recording processing unit that records an image signal subjected to interframe predictive coding processing on the recording medium 12, and reference numeral 14 denotes a display device that displays the image signal subjected to camera signal processing. Reference numeral 13 denotes a display processing unit that displays an image on the display device 14, reference numeral 15 denotes a bus, and reference numeral 16 denotes a system control unit that controls the entire imaging apparatus.

  Hereinafter, the photographing operation performed by the imaging apparatus 200 of the present embodiment configured as described above will be described. First, the lens 1 is driven by a driving pulse from the lens driving unit 2 controlled by the system control unit 16 to form a subject image on the image sensor 3.

  The image sensor 3 captures an image using a drive pulse from the image sensor driver 4 controlled by the system controller 16. The analog image signal output from the image pickup device 3 is removed from the clock synchronization noise by the CDS circuit provided in the AFE 5 and converted into a digital image signal by the A / D converter provided in the AFE 5. Converted.

  The digital image signal output from the AFE 5 is converted into an image signal of luminance color difference by the camera signal processing unit 8. The phase control unit 6 outputs an image signal that has been subjected to phase compensation processing so that the phase of the input signal does not change according to the driving state of the image sensor 3. The contents of the phase compensation processing include a method of performing interpolation processing from information on a plurality of lines, a method of having a memory inside, performing interpolation calculation processing using image information of the previous field, and generating a frame image. it can.

  For example, in FIG. 5, the phase difference between the even field (5-2) and the odd field (5-3) is seen in the original line of all pixels (5-1). Corresponds to one line. Therefore, the average processing of the upper and lower two lines of either field is performed, or the upper and lower two lines of both fields are processed in the reverse direction at 1: 3, and the interpolation operation between the lines of the input signal is performed. Phase compensation can be applied. It is possible to eliminate the phase change by changing the coefficient of the interpolation operation according to the driving method.

  As a method for performing interpolation processing using image information of the previous field, two consecutive fields are interlaced. For this reason, if the image of the previous field is combined with each field, it is possible to perform phase compensation by a method of generating a frame image (5-1) in a pseudo manner.

  The subject detection unit 7 detects a subject from the image signal subjected to the phase compensation processing output from the phase control unit 6, and outputs the subject detection information to the system control unit 16 via the bus 15. The subject detection frame display control unit 9 creates frame information to be superimposed on the display image based on the subject detection information output from the subject detection unit 7.

  The image superimposing unit 10 superimposes a frame based on the frame information output from the subject detection frame display control unit 9 on the input image signal to the display device 14. The display processing unit 13 converts the image signal output from the image superimposing unit 10 into a signal suitable for the display device 14 (for example, an NTSC analog signal) and displays the signal on the display device 14. The recording processing unit 11 compresses and encodes the image signal output from the camera signal processing unit 8, converts it into data suitable for the recording medium 12 (for example, file system data having a hierarchical structure), and records the data on the recording medium 12. The system control unit 16 controls operations of the lens driving unit 2, the image sensor driving unit 4, the camera signal processing unit 8, the recording processing unit 11, the display processing unit 13, and the like.

FIG. 4 is a timing chart showing the operation of the phase controller 6.
In FIG. 4, 4-1 represents a synchronization signal in units of fields. Reference numeral 4-2 represents an output signal from the image sensor, where the even field is E, the odd field is O, and the numbers are serial numbers. It is assumed that the even field and the odd field have different image pickup element driving methods and output images having different phases.

  4-3 is an image signal generated by the phase control unit 6 based on the imaging method, and is generated in the same field as the signal field output from the imaging element 3, for example, E0 ′ from the image signal in the E0 field. Image signals are generated.

  4-4 is subject information detected by the subject detection unit 7. The subject detection unit 7 performs a detection operation from the image signal output from the phase control unit 6 and outputs a detection result after one field. Therefore, the detection result DET_E0 ′ in the E0 ′ field is output.

Next, the processing procedure of this embodiment will be described with reference to the flowchart of FIG.
When the process is started, shooting by the image sensor 3 is started in step S71, and an analog image signal is generated and output.

In step S72, the clock synchronization noise is removed from the analog image signal output from the image sensor 3, and A / D conversion is performed to generate a digital image signal.
Next, proceeding to step S73, the digital image signal generated at step S72 is converted into an image signal of luminance color difference.

  Next, in step S74, phase control processing is performed, and an image signal having the same field as the signal output from the image sensor 3 is generated. Thereby, as described above, for example, the image signal of E0 ′ is generated from the image signal of the E0 field.

  Next, the process proceeds to step S75, where the subject is detected from the image signal generated in step S74. Thus, in the present embodiment, in the same frame, subject detection is performed using image signals having the same phase, so that the detection result is output one field later.

  Next, the process proceeds to step S76, frame information to be superimposed on the display image is calculated from the subject detection information output from step S75, and the calculated frame information is output to the image superimposing unit 10.

  Next, in step S77, the frame is superimposed on the image signal based on the frame information generated in step S76. Next, in step S78, the image signal on which the frame is superimposed is converted into a display signal suitable for the display device 14 (for example, an NTSC analog signal or the like) and displayed on the display device 14. Next, in step S79, it is determined whether or not all processing has been completed. If the result of this determination is not complete, the process returns to step S71 to perform the above-described processing. If the result of determination in step S79 is that the processing has ended, end processing is performed.

  Also in the present embodiment, subject detection is performed even when images with different phases are output by performing the processing as described above, because the driving method of the image sensor 3 is different between the even field and the odd field. It becomes possible not to affect. Thereby, subject detection and display can be stabilized.

(Other embodiments according to the present invention)
Each unit constituting the imaging apparatus according to the above-described embodiment of the present invention can be realized by operating a program stored in a RAM or a ROM of a computer. This program and a computer-readable recording medium recording the program are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowcharts shown in FIGS. 6 and 7) for executing each step in the above-described imaging method is directly or remotely supplied to the system or apparatus. And the case where it is achieved also by reading and executing the supplied program code by the computer of the system or apparatus is included.

  Therefore, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Various recording media can be used as a recording medium for supplying the program. For example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD- R).

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. It can also be supplied by downloading the computer program itself of the present invention or a compressed file including an automatic installation function from a homepage to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, the present invention includes a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer may perform part or all of the actual processing. The functions of the above-described embodiments can be realized.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

DESCRIPTION OF SYMBOLS 1 Lens, 2 Lens drive part, 3 Image sensor, 4 Image sensor drive part, 5 AFE, 6 Phase control part, 7 Subject detection part, 8 Camera signal processing part, 9 Subject detection frame display control part, 10 Image superimposition part, DESCRIPTION OF SYMBOLS 11 Recording processing part, 12 Recording medium, 13 Display processing part, 14 Display apparatus, 15 Bus | bath, 16 System control part, 100 Imaging device

Claims (10)

An image sensor comprising a plurality of pixels;
Drive control means for controlling the drive method of the image sensor;
Subject detection means for recognizing a predetermined subject from image signals output from the image sensor and extracting first subject information;
Subject information display control means for processing the first subject information to generate second subject information and superimposing it on an input image to the image display means;
Phase control means for controlling the phase of an input signal to the subject detection means based on the driving method of the image sensor;
An image pickup apparatus characterized in that a change in phase of an image signal caused by changing a driving method of the image pickup element does not affect subject detection.   The phase control unit is configured to eliminate a change in phase caused by a change in a plurality of driving methods by controlling an output signal when the imaging element is driven by the same driving method to detect a subject. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized.   2. The phase control unit according to claim 1, wherein a phase change caused by a change in a plurality of drive methods is eliminated by performing a phase compensation process of an input signal of an imaging method in accordance with the drive method. Imaging device.   The phase compensation processing performed in the phase control means is based on interpolation between the lines of the input signal, and the phase change is eliminated by changing the coefficient of the interpolation according to the driving method. The imaging apparatus according to claim 3.   4. The imaging apparatus according to claim 3, wherein the phase compensation processing performed in the phase control means eliminates a change in phase by generating a frame image by interpolation between fields of the input signal.   The change in the driving method of the image sensor is to periodically change a combination of pixel addition or thinning so that an output signal becomes an interlace signal. The imaging apparatus of Claim 1.   The first subject information is position / size information of a detected subject, and the second subject information is a frame signal created based on the first subject information. The imaging device according to any one of 1 to 4. A drive control step for controlling the drive method of the image sensor composed of a plurality of pixels;
A subject detection step of recognizing a predetermined subject from the image signal output from the image sensor and extracting first subject information;
A subject information display control step of processing the first subject information to generate second subject information and superimposing it on an input image to the image display device;
A phase control step for controlling the phase of the input signal to the subject detection step based on the driving method of the image sensor;
An imaging method characterized in that a change in the phase of an image signal caused by changing the driving method of the imaging element does not affect subject detection. A drive control step for controlling the drive method of the image sensor composed of a plurality of pixels;
A subject detection step of recognizing a predetermined subject from the image signal output from the image sensor and extracting first subject information;
A subject information display control step of processing the first subject information to generate second subject information and superimposing on the input image to the image display device;
A phase control step for controlling the phase of the input signal to the subject detection step based on the driving method of the image sensor;
A computer program that causes a computer to execute so that a change in phase of an image signal caused by changing a driving method of the image sensor does not affect subject detection.   A computer-readable storage medium storing the computer program according to claim 9.

Images