KR102503442B1 - Electronic device and operating method thereof - Google Patents

Abstract

The present disclosure relates to an electronic device and a control method of the electronic device, and includes a plurality of image sensors including a first image sensor and a second image sensor and electrically connected to the plurality of image sensors, a read control signal and a synchronization signal to the plurality of image sensors, the processor outputs a first read control signal to the first image sensor, receives first data read from the first image sensor, and generates a second read control signal is output to the second image sensor, second data read from the second image sensor is stored in a temporary memory, and based on the output control signal generated between the first read control signal and the next first read control signal, the temporary memory Control to output the second data stored in , and generate merged data by merging the first data and the second data.

Description

Electronic device and control method of the electronic device

Embodiments relate to electronic devices and control methods of the electronic devices, and more specifically, to controlling read times of image signals generated by a plurality of imaging sensors to efficiently use a memory.

An electronic device may process an image acquired through an imaging sensor. In addition, recent electronic devices have reached a stage of mobile convergence that encompasses the functions of other devices. An electronic device may provide a photographing function by including an image sensor in addition to a call function and a message transmission/reception function.

The imaging sensor may perform a function of converting a received optical signal into an electrical signal through pixels. Pixels may be arranged in a pixel array of a set color pattern. For example, the pixels of the imaging sensor may be R (red), G (green), and B (blue) pixels, and the R, G, and B pixels may be arranged in a pixel array of a set color pattern. Additionally, the pixels may be arranged in a pixel array in a color and brightness pattern. For example, the pixels of the image sensor may be R (red), G (green), B (blue), and W (white) pixels, and the R, G, B, and W pixels are arranged in a pixel array of a set pattern. It can be.

One or more image sensors may be provided in an electronic device. The electronic device may synthesize images captured by two or more image sensors to form a single image.

In the case of photographing using a plurality of image sensors, when storing or processing data read from the plurality of image sensors, providing a temporary memory or processing circuit based on the amount of data read from each image sensor is an electronic device of hardware increases the overall cost.

Accordingly, according to various embodiments, an electronic device capable of optimizing a temporary memory capacity and efficiently configuring hardware by adjusting read timings of data read from a plurality of image sensors and a control method of the electronic device are provided. will be.

An electronic device according to an embodiment includes a plurality of image sensors including a first image sensor and a second image sensor; and a processor electrically connected to the plurality of imaging sensors and outputting a read control signal and a synchronization signal to the plurality of imaging sensors. wherein the processor outputs a first read control signal to the first imaging sensor, receives first data read from the first imaging sensor, and transmits a second read control signal to the second imaging sensor; output, store the second data read from the second imaging sensor in a temporary memory, and store the second data stored in the temporary memory based on an output control signal generated between the first read control signal and the next first read control signal; 2 data is output, and merged data is generated by merging the first data and the second data.

In the control method of an electronic device including a plurality of image sensors including a first image sensor and a second image sensor according to another embodiment, the processor outputs a first read control signal to the first image sensor; , receiving first data read from the first image sensor; outputting a second read control signal to the second imaging sensor and storing second data read from the second imaging sensor in a temporary memory; controlling to output second data stored in the temporary memory based on an output control signal generated between the first read control signal and the next first read control signal; and generating merged data obtained by merging the first data and the second data.

and a recording medium recording a program for executing the control method of the electronic device according to another embodiment in a computer.

According to various embodiments as described above, the electronic device can efficiently store data read from a plurality of imaging sensors and generate a synthesized image with a minimum temporary memory capacity.

1 is an external view of an electronic device 100 according to an embodiment.
2 is a diagram of an electronic device 201 within a network environment 200 according to various embodiments.
FIG. 3 is a schematic block diagram of the electronic device 100 shown in FIG. 1 .
FIG. 4 is an exemplary view illustrating an image pickup sensor module 401 in the electronic device 100 shown in FIG. 1 .
FIG. 5 is an exemplary view illustrating pixels included in an image sensor of the electronic device 100 shown in FIG. 1 .
FIG. 6 is an exemplary view illustrating pixels included in an image sensor of the electronic device 100 shown in FIG. 1 .
7 is a diagram for explaining a process of transmitting an image signal in an electronic device according to an exemplary embodiment.
8 is a diagram for explaining a process of transmitting an image signal based on a synchronization signal of a processor in an electronic device according to an embodiment.
9 is a flowchart illustrating a control method of an electronic device according to another embodiment.
10 and 11 are results of synthesizing a plurality of images into a single image in an electronic device according to an exemplary embodiment.
12 is a flowchart illustrating a method of controlling an electronic device according to another embodiment.
13 is a diagram for explaining a process of transmitting an image signal based on a synchronization signal of a processor in an electronic device according to another embodiment.

Hereinafter, various embodiments will be described with reference to the accompanying drawings. However, this is not intended to limit the technology described in this embodiment to a specific embodiment, and it should be understood that it includes various modifications, equivalents, and/or alternatives of this embodiment. In connection with the description of the drawings, like reference numerals may be used for like elements.

In the present embodiment, expressions such as “has”, “can have”, “includes”, or “can include” indicate a corresponding feature (eg, numerical value, function, operation, or component such as a part). Indicates the presence, and does not exclude the presence of additional features.

In this embodiment, expressions such as "A or B", "at least one of A and/and B", or "one or more of A or/and B" may include all possible combinations of the items listed together. there is. For example, "A or B", "at least one of A and B", or "at least one of A or B" includes (1) at least one A, (2) at least one B, Or (3) may refer to all cases including at least one A and at least one B.

Expressions such as "first", "second", "first", or "second" used in this embodiment may modify various elements regardless of order and/or importance, and may refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, a first component may be named a second component without departing from the scope of rights described in this embodiment, and similarly, the second component may also be renamed to the first component.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that any of the above-mentioned elements may be directly connected to the above-mentioned other elements or may be connected through another element (eg, a third element). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), a component and other elements It may be understood that there are no other components (eg, a third component) between the elements.

The expression “configured to (or configured to)” used in this embodiment may be used depending on the situation, for example, “suitable for” or “having the capacity to”. )", "designed to", "adapted to", "made to", or "capable of" . The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the phrase "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or configured) to perform A, B, and C" may include a dedicated processor (eg, embedded processor) to perform the operation, or by executing one or more software programs stored in a memory device. , may mean a general-purpose processor (eg, CPU or application processor (AP)) capable of performing corresponding operations.

Terms used in this embodiment are only used to describe a specific embodiment, and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in this embodiment. Among the terms used in this embodiment, terms defined in general dictionaries may be interpreted as having the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in this embodiment, ideal or excessively formal. not be interpreted in a literal sense. In some cases, even terms defined in this embodiment cannot be interpreted to exclude embodiments of this embodiment.

Electronic devices according to various embodiments of the present embodiment include, for example, a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, Desktop personal computer (laptop personal computer), netbook computer, workstation, server, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical It may include at least one of a device, a camera, or a wearable device. According to various embodiments, the wearable device may be an accessory (eg, watch, ring, bracelet, anklet, necklace, glasses, contact lens, or head-mounted-device (HMD)), fabric or clothing integrated ( For example, it may include at least one of an electronic clothing), a body attachment type (eg, a skin pad or tattoo), or a living body implantable type (eg, an implantable circuit).

In some embodiments, the electronic device may be a home appliance. Home appliances include, for example, televisions, digital video disk (DVD) players, audio systems, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air purifiers, set-top boxes, and home automation controls. It may include at least one of a home automation control panel, a security control panel, a TV box, a game console, an electronic dictionary, an electronic key, a camcorder, or an electronic photo frame.

In another embodiment, the electronic device may include various types of medical devices (e.g., various portable medical measuring devices (blood glucose meter, heart rate monitor, blood pressure monitor, body temperature monitor, etc.), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), CT (computed tomography), imager, or ultrasonicator, etc.), navigation device, GNSS (global navigation satellite system), EDR (event data recorder), FDR (flight data recorder), automotive infotainment ) devices, marine electronics (e.g. marine navigation systems, gyrocompasses, etc.), avionics, security devices, automotive head units, industrial or home robots, automatic teller's machines (ATMs) of financial institutions , point of sales (POS) in stores, or internet of things devices (e.g. light bulbs, sensors, electric or gas meters, sprinklers, smoke alarms, thermostats, street lights, toasters) , Exercise equipment, hot water tank, heater, boiler, etc.) may include at least one.

According to some embodiments, the electronic device may be a piece of furniture or a building/structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (eg, Water, electricity, gas, radio wave measuring devices, etc.) may include at least one. In various embodiments, the electronic device may be one or a combination of more than one of the various devices described above.

An electronic device according to some embodiments may be a flexible electronic device. Also, the electronic device according to the present embodiment is not limited to the aforementioned devices, and may include new electronic devices according to technological development.

Hereinafter, an electronic device according to various embodiments will be described with reference to the accompanying drawings. In this document, the term user may refer to a person using an electronic device or a device using an electronic device, such as an artificial intelligence electronic device.

1 is an external view of an electronic device 100 according to an embodiment.

Referring to FIG. 1 , the electronic device 100 may be implemented as devices for various purposes. For example, as described above, the electronic device 100 may be implemented as a mobile phone, a smart phone, a laptop computer, a tablet device, etc., but is not limited thereto.

Referring to (a) of FIG. 1 , a display 110 may be installed on the front surface 101 of the electronic device 100 . A speaker 120 for receiving the voice of the other party may be installed above the display 110 . A microphone 130 may be installed below the display 110 to transmit the voice of the user of the electronic device 100 to the other party.

According to an embodiment, components for performing various functions of the electronic device 100 may be disposed around where the speaker 120 is installed. Components may include at least one sensor module 140 . The sensor module 140 may include, for example, at least one of an illuminance sensor (eg, an optical sensor), a proximity sensor, an infrared sensor, and an ultrasonic sensor. The component may also include a camera 150 . According to one embodiment, the part may include an LED indicator 160 for recognizing state information of the electronic device 100 to a user.

The electronic device 100 may capture an object using the camera 150 . 1(b) is a side view of the electronic device 100 . Referring to (b) of FIG. 1 , the electronic device 100 may include another camera 150 and 150'. However, it is not limited thereto, and may include more cameras. The cameras 150 and 150' of the electronic device 100 may form various angles of view. The angle of view may be, for example, 30 degrees, 50 degrees, 90 degrees, or 180 degrees. The electronic device 100 may generate images captured by the cameras 150 and 150', respectively, or may generate a single image by synthesizing the captured images. For example, if the cameras 150 and 150' have a field of view of 180 degrees, the electronic device 100 may generate an image having a field of view of 360 degrees. A camera capable of capturing a 360-degree field of view may be referred to as an omnidirectional camera or a 360-degree camera, and the electronic device 100 according to the embodiment is not limited to the term, and includes a plurality of images captured from a plurality of cameras or imaging sensors. It may be a camera capable of compositing.

2 is a diagram of an electronic device 201 within a network environment 200 according to various embodiments. The electronic device 101 may include the electronic device 100 of FIG. 1 .

Referring to FIG. 2 , an electronic device 201 may include a bus 210, a processor 220, a memory 230, an input/output interface 250, a display 260, and a communication interface 270. In some embodiments, the electronic device 201 may omit at least one of the components or may additionally include other components.

The bus 210 may include circuitry that couples the components 210-270 to each other and communicates between the components, eg, conveying control messages and/or data.

The processor 220 is one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), and an image signal processor. may contain more than The processor 220 may, for example, execute calculations or data processing related to control and/or communication of at least one other component of the electronic device 201 .

Memory 230 may include volatile and/or non-volatile memory. The memory 230 may store, for example, commands or data related to at least one other component of the electronic device 201 . According to one embodiment, the memory 230 may store software and/or programs 240 . Program 240 may include, for example, kernel 241, middleware 243, application programming interface (API) 245, and/or application program (or "application") 147, etc. can include At least a part of the kernel 241, middleware 243, or API 245 may be referred to as an operating system (OS).

Kernel 241, for example, provides system resources (eg, middleware 243, API 245, or application program 247) used to execute operations or functions implemented in other programs (eg, middleware 243, API 245, or application program 247). : The bus 210, the processor 220, or the memory 230, etc.) can be controlled or managed. In addition, the kernel 241 may provide an interface capable of controlling or managing system resources by accessing individual components of the electronic device 201 from the middleware 243, API 245, or application program 247. can

The middleware 243 may perform an intermediary role so that, for example, the API 245 or the application program 247 communicates with the kernel 241 to exchange data.

Also, the middleware 243 may process one or more job requests received from the application program 247 according to priority. For example, the middleware 243 may use system resources (eg, the bus 210, the processor 220, or the memory 230, etc.) of the electronic device 201 for at least one of the application programs 247. You can give priority. For example, the middleware 243 may perform scheduling or load balancing of the one or more task requests by processing the one or more task requests according to the priority assigned to the at least one task.

The API 245 is an interface for the application 247 to control functions provided by the kernel 241 or the middleware 243, for example, file control, window control, image processing, or text. It may include at least one interface or function (eg, command) for control.

The input/output interface 250 may function as an interface capable of transferring commands or data input from a user or other external device to other component(s) of the electronic device 201 . Also, the input/output interface 250 may output commands or data received from other component(s) of the electronic device 201 to a user or other external device.

The display 260 may be, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or It may include a microelectromechanical systems (MEMS) display, or an electronic paper display. The display 260 may display, for example, various contents such as text, images, videos, icons, or symbols to the user. The display 260 may include a touch screen, and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a part of the user's body.

The communication interface 270 establishes communication between the electronic device 201 and an external device (eg, the first external electronic device 202 , the second external electronic device 204 , or the server 206 ). can For example, the communication interface 270 may be connected to the network 262 through wireless or wired communication to communicate with an external device (eg, the second external electronic device 204 or the server 206).

Wireless communication is, for example, a cellular communication protocol, for example, long-term evolution (LTE), LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal At least one of a mobile telecommunications system, WiBro (Wireless Broadband), GSM (global system for mobile communications), and the like may be used. Wireless communication may also include, for example, short-range communication 264 . The short-range communication 264 may include, for example, at least one of wireless fidelity (WiFi), Bluetooth, near field communication (NFC), and global navigation satellite system (GNSS). GNSS is a global positioning system (GPS), Glonass (Global Navigation Satellite System), Beidou Navigation Satellite System (hereinafter referred to as “Beidou”) or Galileo, the European global satellite-based navigation system, depending on the region of use or bandwidth. may contain at least one. Hereinafter, in this document, “GPS” may be used interchangeably with “GNSS”. Wired communication may include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). The network 262 may be any of a telecommunications network, for example, a computer network (eg, a local area network (LAN) or a wide area network (WAN)), the Internet, or a telephone network. may contain at least one.

Each of the first and second external electronic devices 202 and 204 may be the same as or different from the electronic device 201 . According to one embodiment, server 206 may include a group of one or more servers. According to various embodiments, all or part of operations executed in the electronic device 201 may be executed in one or more electronic devices (eg, the electronic devices 202 and 204 or the server 206). According to this, when the electronic device 201 needs to perform a certain function or service automatically or upon request, the electronic device 201 instead of or in addition to executing the function or service by itself, at least some of the functions or services related thereto A function may be requested from another device (eg, the electronic device 202 or 204 or the server 206) The other electronic device (eg, the electronic device 202 or 204 or the server 206) may request the requested function Alternatively, an additional function may be executed and the result may be transmitted to the electronic device 201. The electronic device 201 may provide the requested function or service by processing the received result as it is or additionally. For example, cloud computing, distributed computing, or client-server computing technologies may be used.

According to an embodiment, the electronic device 101 and the first and second external electronic devices 202 and 204 use the above-described communication methods to transmit images captured using cameras in each electronic device to one electronic device. and can be combined into one image. For example, an image having a 360-degree field of view may be generated by synthesizing a 180-degree field of view image captured by the electronic device 201 and a 180-degree field of view image captured by the second electronic device 202 .

FIG. 3 is a schematic block diagram of the electronic device 100 shown in FIG. 1 .

Referring to FIG. 3 , the electronic device 100 may include a processor 310, a temporary memory 320, a sensor A 330, and a sensor B 340. The electronic device 200 according to an embodiment may be implemented with more or fewer components than the components shown in FIG. 3 , since the components shown in FIG. 3 are not essential. For example, the electronic device 100 may include, for example, a touch panel, a physical key, a proximity sensor, and a bio sensor as an input module, and may include a memory, an A/D (Analog/Digital) converter, a power supply unit, and the like. It can be configured including configuration. Here, the processor 310 refers to a processor that controls the imaging sensors A and B 330 and 340, receives image data, and performs image processing.

The imaging sensor A 330 and the imaging sensor B 340 may receive light incident through a lens (not shown) and convert it into an electrical signal. Image sensor A 330 may be referred to as a first image sensor, and image sensor B 340 may be referred to as a second image sensor. The first and second image sensors 330 and 340 may include pixel sensors capable of detecting at least two pieces of mixed color information, for example, a white (w) pixel or a bright pixel. For example, the first and second image sensors 330 and 340 include a pixel array in which red (R), green (G), and blue (B) pixel sensors and white (W) pixel sensors are arranged in a set pattern. can do.

The temporary memory 320 may temporarily store image signals generated and read by the second image sensor 340 . The temporary memory 320 may be provided separately from the processor 310 and may be a buffer memory or a line memory. Image signals may be expressed and stored in the form of general data transmitted between electronic devices. The temporary memory 320 may temporarily store only image signals read from one of the two image sensors. As shown in FIG. 3 , the image signal generated by the first imaging sensor 330 is directly transmitted to the processor 310 without going through the temporary memory 320, and the image signal generated by the second imaging sensor 340 Is temporarily stored in the temporary memory 320 and then transmitted to the processor 310. The processor 310 controls the image signal stored in the temporary memory 320 to be received when the reception of the image signal read from the first image sensor 330 ends. Since the processor 310 must not receive the next image signal read from the first imaging sensor 330 while receiving the image signal from the temporary memory 320, the read control signal output to the first imaging sensor 330 A blanking period may be inserted so that image signals received from the first image sensor 330 and the second image sensor 340 do not overlap. The setting of the blanking section will be described later with reference to FIG. 8 . As shown in FIG. 3 , the electronic device 100 according to the present embodiment uses a temporary memory for storing image signals output from two imaging sensors to store image signals output from one imaging sensor, for example, one horizontal line. Since it is implemented only with the line memory corresponding to the data of , the number of line memories required can be reduced. Also, image signals input from two image sensors may be processed like image signals input from one image sensor. In the case of a camera including a plurality of sensors and lenses, such as a 3D camera, a dual camera, or a 360-degree omnidirectional camera using the electronic device according to the present embodiment, exposure starts and ends at exactly the same time, so that even when shooting a fast object, the user can You can get the effect of shooting with a single sensor without any inconvenience.

Although it has been described that the electronic device according to the embodiment has two imaging sensors, even when N (N is an integer of 2 or more) or more is used, the N or more imaging sensors are simultaneously or individually controlled as a synchronization signal and a control signal. It can receive data and generate a merged image at the same time using N-1 line memory.

The processor 310 may control the operation of the above-described components. For example, the processor 310 may generate a synchronization signal to generate image signals at regular times or at regular time intervals. The processor 310 may control the generated image signals to be sequentially transmitted to the temporary memory 320 . For example, some of the signals generated by the imaging sensor A 330 may first be transmitted to the temporary memory 320, and then some of the signals generated by the imaging sensor B 340 may be transmitted to the temporary memory 320. there is.

The electronic device 100 may separately provide an image processing unit (not shown) that processes image signals, and may include it inside the processor 310 . Hereinafter, it is assumed that the processor 310 processes an image. The processor 310 may perform various post-processing using the image signals received through the temporary memory 320 . For example, the processor 310 may perform gain adjustment or signal processing for shaping a waveform with respect to the received image signal.

The processor 310 may perform a process of removing noise from the image signal. For example, the processor 310 may perform gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, and the like. Signal processing for image quality improvement may be performed.

Each of the imaging sensors 330 and 340 may be included in an imaging sensor module. 4 illustrates an imaging sensor module in an electronic device according to an exemplary embodiment.

Referring to FIG. 4 , an imaging sensor module 401 according to various embodiments includes a cover member 450, a lens 410, a lens barrel 420, a coil unit 430, a magnet unit 440, and a base ( 470), an imaging sensor 460, and a circuit board 480.

The cover member 450 may form an outside of the image pickup sensor module 401 . The cover member 450 may protect various components disposed in the inner space.

The lens 410 may include a plurality of lenses 410 . For example, a lens that moves for auto focus and a lens that moves for zooming may be included. An image received from the lens 410 may be transmitted to the image sensor 460 .

The lens barrel 420 may accommodate the lens 410 . A coil unit 430 may be disposed outside the lens barrel 420 . The magnet part 440 may be disposed at a position corresponding to the coil part 430 . The magnet unit 440 may be disposed to face the coil unit 430 . According to various embodiments of the present disclosure, the electronic device 100 uses an optical image stabilizer (OIS) driving module (not shown) to generate a lens ( 410) to correct the user's shaking (optical image stabilizer, OIS).

For example, in the electronic device 102, power is applied to the coil unit 430, and the lens 410 is moved by an interaction between an electromagnetic field generated from the coil unit 430 and a magnetic field generated from the magnet unit 440. can move Through this, the electronic device 102 can detect the user's hand shake and move the lens 410 in the opposite direction of the shake, thereby preventing image blur. The electronic device 102 according to an embodiment may detect the user's hand shake and move the image sensor 460 in the opposite direction of the hand shake to prevent image blur.

The base 470 may be coupled to the cover member 450 . The base 470 may support the lower side of the cover member 450 . An infrared cut filter may be additionally disposed on the base 470 at a position corresponding to the imaging sensor 460 . The base 470 may function as a sensor holder to protect the imaging sensor 460 .

The imaging sensor 460 may be disposed on the circuit board 480 . The image sensor 460 may be electrically connected to the circuit board 480 by wire bonding or may be flip bonded using an electrically conductive paste.

The circuit board 480 may include a plurality of circuit patterns (not shown) and transmit a signal converted using the imaging sensor 460 to the processor 310 .

The image sensor 460 may include a pixel array in which color pixel sensors (eg, R, G, and B pixels) and white pixels (eg, W pixels) are arranged in a set pattern. The pixel array converts an optical image signal of an external object incident through the lens 410 into an electrical image signal. 5 illustrates pixels included in an image sensor of an electronic device according to various embodiments.

As shown in FIG. 5 , each pixel 501 may convert an optical signal into an electrical signal. Each pixel 501 may include at least one micro lens (not shown), at least one color filter 510 , and at least one photodiode 520 .

The micro lens may condense light incident from the outside.

The color filter 510 includes at least one of a red filter, a green filter, a blue filter, a white filter, a cyan filter, a magenta filter, and a yellow filter. may contain one.

The photodiode 650 may convert an optical signal into an electrical signal.

6 is an exemplary diagram for describing pixels included in an image sensor of an electronic device according to various embodiments.

As shown in FIGS. 5 and 6 , the red filter may pass light in a red wavelength band. The green filter may pass light in a green wavelength band. The blue filter may pass light in a blue wavelength band. The white filter can pass light of all wavelength bands in the visible light region. The cyan filter may pass light in a green wavelength band and a blue wavelength band. The magenta filter may pass light in a red wavelength band and a blue wavelength band. The yellow filter may pass light in a red wavelength band and a green wavelength band.

7 is a diagram for a situation in which an image signal is transmitted to a temporary memory in an electronic device according to an embodiment. In the following description, it is assumed that the imaging sensor includes 8x8 pixels.

Referring to FIG. 7 , the processor 310 may capture an object by controlling the imaging sensor module 401 shown in FIG. 4 . The processor 310 may control the plurality of imaging sensors 710 and 720 to capture images almost simultaneously using a synchronization signal or the like. The imaging sensors 710 and 720 may change the received optical signal into an image signal. To this end, the image sensors 710 and 720 transmit charge generated in each pixel 711 and 721, that is, an image signal, to the processor 310. In this case, the image signal read from the imaging sensor A 710 is directly transmitted to the processor 310, and the image signal read from the imaging sensor B 720 is transmitted to the temporary memory 730.

The imaging sensors 710 and 720 may transmit image signals to the processor 310 and the temporary memory 730 line by line. Specifically, the imaging sensor A 710 transmits the first line 715 to the processor 310 . Meanwhile, the image sensor B 720 transmits the first line 725 to the temporary memory 730 . Due to this, there is only a space accommodating only the data of the first line 725 of the imaging sensor B 720, so that the received data can be smoothly output to the processor 310. The temporary memory 730 stores the data of the first line read from the image sensor B 720 and stored when the processor 310 finishes receiving the data 715 of the first line read from the image sensor A 710. 725 to the processor 310. Accordingly, the electronic device according to the embodiment has only one temporary memory, for example, a line memory capable of storing data of one line, for example, 8 pixels, of the imaging sensor B 720, but the processor 310 can generate the same data. A merged image may be generated by receiving image signals read from each image sensor at the time. Since the processor 310 cannot receive the next image signals a9-a16 from the image sensor A 710 while receiving the image signal 725 from the temporary memory 730, the image signal 725 from the image sensor A 710 An interval before outputting the next horizontal synchronizing signal or the next read control signal for controlling image signal reading is set as a blanking interval.

A detailed process of transmitting image signals from the imaging sensors 710 and 720 to the processor 310 via the temporary memory 730 will be described later with reference to FIG. 8 .

The processor 310 may connect the first line 715 received from the imaging sensor A 710 and the first line 725 received from the imaging sensor B 720 to form one line. By repeating this process, the processor may generate one image by synthesizing the image generated by the image sensor A 710 and the image generated by the image sensor B 720 .

8 is a diagram illustrating a process of transmitting an image signal based on a synchronization signal of a processor in an electronic device according to an embodiment.

Referring to FIGS. 7 and 8 , the processor 310 may control each of the imaging sensors 710 and 720 using a vertical synchronization signal 810 and a horizontal synchronization signal 812 . The vertical synchronization signal 810 is a signal for synchronizing each frame, and the horizontal synchronization signal 812 is a signal for synchronizing each line included in a frame. Here, the vertical synchronizing signal 810 and/or the horizontal synchronizing signal 812 may be read control signals that control data reading from each imaging sensor 810 . Here, the processor 310 may simultaneously or sequentially output read control signals output to the imaging sensor A 710 and the imaging sensor B 720 . Also, the processor 310 may synchronize (sync) or asynchronously (async) output each read control signal to each imaging sensor.

Based on the vertical synchronization signal 810 and the horizontal synchronization signal 812 , the imaging sensor A 710 transmits an image signal or data 715 corresponding to the first line to the processor 310 .

At time T1, when the imaging sensor A 710 outputs the first line of data 715, the processor 310 starts receiving the first line of data 715. Similarly, the imaging sensor B 720 outputs the data 725 of the first line to the temporary memory 730, and the data 725 of the first line is sequentially stored in the temporary memory 730.

At time T2, reception of the data 715 from the imaging sensor A 710 ends, and data 725 is output from the temporary memory 730 to the processor 310. Further, the processor 310 controls not to receive second line data a9 - a16 from the image sensor 710 while receiving the data 725 from the temporary memory 310 . To this end, the timing of the horizontal synchronization signal 812 for controlling reception of the second line data (a9 to a16) from the imaging sensor A 710 is controlled. That is, the interval from T2, when the data 725 is received from the temporary memory 814, to T3, when the second line data (a9 to a16) is received from the imaging sensor A 710, is set as a blanking interval. do.

After the processor 310 receives all of the data 715 from the imaging sensor A 710 and then receives all of the data 725 from the imaging sensor B 720 through the temporary memory 730, the processor 310 receives the data (a9-a16) of the second line from the image sensor A (710) in the next horizontal synchronization signal, and the data (b9-b16) read from the image sensor B (720) is stored in the temporary memory 730 again. Saved. Further, the processor 310 receives the data b9 to b16 stored in the temporary memory 730 at the point of completion of receiving the data a9 to a16 from the imaging sensor A 710 .

As a result, as described above, the processor 310 inserts an arbitrary blanking section into the reading section of the data received from the imaging sensor A 710, thereby using only the line memory corresponding to one line of the imaging sensor B 720. , data output from the two imaging sensors 710 and 720 can be smoothly received.

As described above with reference to FIG. 7 , the processor 310 receives both the data 715 of the first line of the imaging sensor A 710 and the data 725 of the first line of the imaging sensor B 720, A line or a horizontal line of an imaging device may be connected by a single line. In addition, a single image may be generated by accumulating and combining all the lines coming from the image sensors 710 and 720 .

9 is a flowchart illustrating a control method of an electronic device according to another embodiment.

In step 910, the electronic device 100 may generate a photographing signal by an event such as a user input. When a photographing signal is generated, the electronic device 100 may photograph an object by operating, for example, the imaging sensor module 401 . In step 920, the electronic device 100 captures an object using the image sensor A 710 and the image sensor B 720.

In step 930, the electronic device 100 directly transmits the first line 715 of the imaging sensor A 710 to the processor 310 without passing through the temporary memory 730.

In step 940, the processor 310 starts storing the first line of imaging sensor B in temporary memory. Here, steps 930 and 940 may be performed simultaneously or with a time difference.

In step 950, when the processor 310 receives the last data of the first line of the imaging sensor A 710, the first data of the first line of the imaging sensor B 720 stored in the temporary memory 730 is stored in the processor 310. Send to (310).

In step 960, the processor 310 merges the first line of image sensor A and the first line of image sensor B into one continuous line. The processor 310 generates merged image data of one horizontal line after receiving all data of the first line or horizontal line of the image sensor A 710 and the image sensor B 720 . Also, selectively, the processor 310 receives all the data of the first line or the horizontal line of the imaging sensor A 710 and the imaging sensor B 720, and after receiving all the data of the second and third lines, Image data merged frame by frame may also be created.

The electronic device 100 may generate one synthesized image by repeating these steps with respect to lines transmitted from the respective imaging sensors 710 and 720 .

In the control method of an electronic device according to an embodiment, hardware may be configured with only a temporary memory corresponding to one line of one imaging sensor for data read from a plurality of imaging sensors, and timing of synchronization signals of the imaging sensors may be adjusted. Thus, it is possible to efficiently store data read from a plurality of imaging sensors and generate a synthesized image with a minimum temporary memory capacity.

10 is a result of combining two images into one image in an electronic device according to an embodiment.

Referring to (a) of FIG. 10 , image 1 (1010) may be an image generated by the imaging sensor A (710). The imaging sensor A 710 may be, for example, an imaging sensor included in the camera 150 positioned on the front of the electronic device 100 in FIG. 1 . Image 1 (1010) may be, for example, an image having an angle of view of 180 degrees.

Image 2 1020 may be an image generated by imaging sensor B 720 . The imaging sensor B 720 may be, for example, an imaging sensor included in the camera 150' located on the opposite side of the front of the electronic device 100 in FIG. 1 . Image 2 (1020) may also be an image having an angle of view of 180 degrees.

Referring to (b) of FIG. 10 , image 3 (1030) may be a synthesized image of image 1 (1010) and image 2 (1020) described above. Image 1 (1010) and Image 2 (1020) each have a 180-degree angle of view, so image 3 (1030), a combination of the two images, can express a 360-degree angle of view.

If an object is photographed using the electronic device 100 to which this embodiment is applied, the user can easily obtain an image having a 360-degree field of view with one photograph.

11 is a result of combining four images into one image in an electronic device according to another embodiment.

Referring to (a) of FIG. 11 , image 1 (1110) is an image generated by the first imaging sensor, and may be, for example, an image having a 90-degree field of view. Image 2 1120 is an image generated by the second imaging sensor, and may be, for example, an image having a 90-degree field of view. Image 3 1130 is an image generated by the third imaging sensor, and may be, for example, an image having a 90-degree field of view. Image 4 1140 is an image generated by the fourth image sensor, and may be, for example, an image having a 90-degree field of view.

Referring to (b) of FIG. 11 , image 5 (1150) may be an image synthesized from image 1 (1310), image 2 (1320), image 3 (1330), and image 4 (1340) described above. Since each image has a 90-degree angle of view, image 5 (1150), which is a combination of four images, can express a 360-degree angle of view.

If an object is photographed using the electronic device 100 to which this embodiment is applied, the user can easily obtain an image having a 360-degree field of view with one photograph.

12 is a flowchart illustrating a method of controlling an electronic device according to another embodiment.

Referring to FIG. 12 , in step 1200, the processor outputs a first read control signal to the first imaging sensor and receives the read first data.

In operation 1210, a second read control signal is output to the second imaging sensor, and the read second data is stored in a temporary memory.

Steps 1200 and 1210 may be performed simultaneously or sequentially, and the first read control signal and the second read control signal may be a horizontal synchronization signal output to each imaging sensor or a sensor control signal corresponding to the horizontal synchronization signal.

In operation 1220, the processor controls the second data stored in the temporary memory to be output based on the output control signal generated between the first read control signal and the next first read control signal. Here, a blanking section is inserted between a first read control signal for reading data of a first line of the first image sensor and a next first read control signal for reading data of a second line of the first image sensor. Accordingly, the processor may receive the second data stored in the temporary memory immediately after receiving the first data from the first image sensor, and may not receive data from the first image sensor while receiving the second data. The length of the blanking section may be variably set in consideration of the number of imaging sensors and the size of one line of the imaging sensors.

In operation 1230, the processor generates merged data obtained by merging the first data and the second data.

The control method of an electronic device according to an embodiment can minimize the use of temporary memory and set data input and output clocks to the same data simply by providing a certain blanking section in the data read timing of any one of a plurality of image sensors. Even if this is done, it does not cause inconvenience in synthesizing a plurality of image data in the processor.

13 is a diagram for a situation in which an image signal is transmitted to a temporary memory in an electronic device according to another embodiment. Comparing the embodiment described with reference to FIG. 13 and the embodiment of FIG. 8 , the electronic device according to the embodiment of FIG. 13 does not include a separate temporary memory. Here, data may be sequentially output to the processor 310 without a temporary memory by adjusting read timings of the image sensor A 710 and the image sensor B 720 .

Referring to FIG. 13 , the processor 310 controls each of the imaging sensors 710 and 720 using a vertical synchronization signal 810 and a horizontal synchronization signal 812 .

At time T1, when the imaging sensor A 710 outputs the first line of data (a1 to a8), the processor 310 starts receiving the first line of data 715. At the timing when the processor 310 finishes receiving the first line of data (a1 to a8), the image sensor B 720 starts outputting the first line of data (b1 to b8) to the processor. That is, the sensor output timings of the imaging sensor A 710 and the imaging sensor B 720 are formed with a time difference of T2-T1. Also, the processor 310 controls not to receive data from the imaging sensor A 710 while receiving the data b1 to b8 from the imaging sensor B 720 . To this end, the timing of the horizontal synchronization signal 812 for controlling reception of the second line data (a9 to a16) from the imaging sensor A 710 is controlled. That is, the interval from T2, when the data 725 is received from the temporary memory 814, to T3, when the second line data (a9 to a16) is received from the imaging sensor A 710, is set as a blanking interval. do.

After the processor 310 receives all of the data (a1-a8) from the imaging sensor A (710) and subsequently receives the data (b1-b8) from the imaging sensor B (720), the processor 310 performs the next horizontal sync When the second line of data (a9-a16) is received from the image sensor A (710) as a signal and the reception of the data (a9-a16) from the image sensor A (710) ends, from the image sensor B (720) Data of the next line (b9-b16) is received.

As a result, as described above, the processor 310 controls the data read timings of the image sensor A 710 and the image sensor B 720 with a time difference, and the read control signal of the image sensor A 710 receives a constant By setting the section blanking section, data can be received from the imaging sensor A 710 and the imaging sensor B 720 without overlapping without using a temporary memory, and merged image data can be generated at the same time.

At least some of devices (eg, modules or functions thereof) or methods (eg, operations) according to various embodiments of the present disclosure are, for example, computer-readable storage media in the form of program modules. It can be implemented as a command stored in . When the command is executed by a processor (eg, the processor 120), the one or more processors may perform a function corresponding to the command. The computer-readable storage medium may be, for example, the memory 130 .

Computer-readable recording media include hard disks, floppy disks, magnetic media (eg magnetic tape), optical media (eg CD-ROM (compact disc read only memory), DVD ( digital versatile disc), magneto-optical media (such as floptical disk), hardware devices (such as read only memory (ROM), random access memory (RAM), or flash memory, etc.) ), etc. In addition, the program command may include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The above-described hardware device may include various It may be configured to act as one or more software modules to perform the operations of an embodiment, and vice versa.

A module or program module according to various embodiments may include at least one or more of the aforementioned components, some may be omitted, or additional components may be further included. Operations performed by modules, program modules, or other components according to various embodiments may be executed in a sequential, parallel, repetitive, or heuristic manner. Also, some actions may be performed in a different order, omitted, or other actions may be added. And the embodiments disclosed in this embodiment are presented for explanation and understanding of the disclosed technical content, and do not limit the scope of the technology described in this embodiment. Therefore, the scope of this embodiment should be construed as including all changes or various other embodiments based on the technical idea of this embodiment.

100: electronic device
310: processor
320: temporary memory
330: imaging sensor A
340: imaging sensor B

Claims (20)

In electronic devices,
a plurality of imaging sensors including N (where N is an integer of 2 or greater) number of imaging sensors;
a temporary memory including N-1 line memories and electrically connected to second to N th image sensors among the N image sensors; and
a processor electrically connected to a first image sensor and the temporary memory and outputting a read control signal and a synchronization signal to the plurality of image sensors; including,
the processor,
A first read control signal is output to the first imaging sensor, and first data read from the first imaging sensor is received, wherein the first data is selected from among pixels included in the first imaging sensor on a horizontal axis line. contains data that can be generated;
An additional read control signal is output to the second to Nth image sensors, second to Nth data read from the second to Nth image sensors is stored in a temporary memory, and the second to Nth data is stored in a temporary memory. includes data that can be generated on a horizontal axis line among pixels included in the second to Nth image sensors;
When the reception of the first data is terminated, the second to Nth data stored in the temporary memory are sequentially output based on an output control signal generated between the first read control signal and the next first read control signal; , An electronic device generating merged data obtained by merging the first data and the second to Nth data into one horizontal axis line. According to claim 1,
The temporary memory is located inside the electronic device separately from the processor. delete delete According to claim 1,
the processor,
An electronic device that controls to output the second through Nth data stored in the temporary memory according to timing when the reception of the first data is terminated. According to claim 5,
the processor,
An electronic device inserting a blanking period in a period between the first read control signal and the next first read control signal. According to claim 1,
wherein N is 2,
The first imaging sensor is included in a first imaging sensor module having a field of view of 180 degrees, and the second imaging sensor is included in a second imaging sensor module having a field of view of 180 degrees;
the processor,
An electronic device generating an image having an angle of view of 360 by merging an image generated by the first image sensor module and an image generated by the second image sensor module. delete According to claim 1,
A speed at which the second to Nth data stored in the temporary memory is output according to the output control signal is the same as a speed at which the second to Nth data stored in the temporary memory are read from a plurality of image sensors according to the first read control signal and the additional read control signal. . According to claim 1,
the processor,
An electronic device that outputs the first read control signal and the additional read control signal to the N imaging sensors simultaneously or with a time difference. A method for controlling an electronic device including a plurality of image sensors including N (where N is an integer of 2 or more) image sensors, the method comprising:
outputting a first read control signal to a first imaging sensor and receiving first data read from the first imaging sensor, wherein the first data is generated on a horizontal axis line among pixels included in the first imaging sensor; including data that can be;
An additional read control signal is output to second to Nth image sensors, second to Nth data read from the second to Nth image sensors is stored in a temporary memory, and the second to Nth data is stored in the second to Nth image sensor. Among the pixels included in the 2nd to Nth image sensors, data that can be generated on a horizontal axis line is included, the temporary memory includes N-1 line memories, and the second to Nth image sensors among the N image sensors Step that is electrically connected to the sensor; and
When the reception of the first data is finished, the second to Nth data stored in the temporary memory are sequentially outputted based on an output control signal generated between the first read control signal and the next first read control signal. controlling; and
and generating merged data obtained by merging the first data and the second to Nth data into one horizontal axis line. According to claim 11,
The temporary memory is located inside the electronic device separately from the processor. delete delete delete According to claim 11,
The method of claim 1 , wherein the second to Nth data stored in the temporary memory are controlled to be output according to a timing at which reception of the first data is terminated. ◈Claim 17 was abandoned when the registration fee was paid.◈ According to claim 16,
The control method of the electronic device further comprising inserting a blanking period between the first read control signal and the next first read control signal. ◈Claim 18 was abandoned when the registration fee was paid.◈ According to claim 11,
wherein N is 2,
The first imaging sensor is included in a first imaging sensor module having a field of view of 180 degrees, and the second imaging sensor is included in a second imaging sensor module having a field of view of 180 degrees;
The control method of the electronic device further comprising generating an image having an angle of view of 360 by merging the image generated by the first image sensor module and the image generated by the second image sensor module. ◈Claim 19 was abandoned when the registration fee was paid.◈ According to claim 11,
A speed at which the second to Nth data stored in the temporary memory is output according to the output control signal is the same as a speed at which the second to Nth data stored in the temporary memory are read from a plurality of image sensors according to the first read control signal and the additional read control signal. control method. ◈Claim 20 was abandoned when the registration fee was paid.◈ A computer-readable recording medium in which a program capable of executing the control method of an electronic device according to any one of claims 11, 12, and 16 to 19 in a computer is recorded.

Images