WO2014198632A1

WO2014198632A1 - Refrigeration appliance comprising a camera module

Info

Publication number: WO2014198632A1
Application number: PCT/EP2014/061706
Authority: WO
Inventors: Gerhard Wetzl; Wolfgang Schramm; Jens Wichert
Original assignee: BSH Bosch und Siemens Hausgeräte GmbH
Priority date: 2013-06-14
Filing date: 2014-06-05
Publication date: 2014-12-18
Also published as: DE102013211098A1

Abstract

The invention relates to a refrigeration appliance (100) comprising a camera module (101) for detecting image data of an item to be refrigerated in a refrigerating compartment (105) of the refrigeration appliance (100), a processing device (119) for processing the image date and a data bus (121) for transferring the image data from the camera module (101) to the processing device (119).

Description

REFRIGERATOR WITH A CAMERA MODULE

The present invention relates to a refrigerator with a camera module.

It is the object underlying the invention to provide a refrigeration device in which data from one or more camera modules can be processed efficiently in the refrigerator.

This object is achieved by the subject matter having the features of the independent claim. Advantageous embodiments of the invention are the subject of the figures, the description and the dependent claims.

According to one aspect of the invention, the object is achieved by a refrigeration device, which is a camera module for capturing image data of a refrigerated goods in a refrigeration compartment of the

Refrigeration device, a processor device for processing the image data and a data bus for transmitting the image data from the camera module to the processor device. As a result, for example, the technical advantage is achieved that data from one or more camera modules can be transferred in a rapid manner to a central processor device within the refrigeration device. Under a refrigeration device is in particular a household refrigeration appliance understood, ie a

Refrigeration appliance, which is used for housekeeping in households or in the catering sector, and in particular serves food and / or drinks at certain

Store temperatures such as a refrigerator, a freezer, a fridge freezer, a freezer or a wine fridge.

In an advantageous embodiment of the refrigeration device, the data bus is a serial data bus. As a result, for example, the technical advantage is achieved that a low cabling effort for connecting the camera modules arises. In a further advantageous embodiment of the refrigeration device, the data bus is guided by a hinge in a door of the refrigeration device. As a result, for example, the technical advantage is achieved that the data bus is protected in the region of the hinge from damage. In a further advantageous embodiment of the refrigeration device, the refrigeration device comprises a plurality of camera modules, which are connected to the data bus. This will

For example, the technical advantage achieved that can produce multiple views of the goods to be cooled or monitor several cold shelves. In a further advantageous embodiment of the refrigerator, the camera module is assigned a unique address, via which the camera module by means of the data bus

is controllable. As a result, for example, the technical advantage is achieved that the data can be transmitted from and to a plurality of camera modules. In a further advantageous embodiment of the refrigeration device comprises the

Processor device an interface for transmitting the image data to an external network. As a result, for example, the technical advantage is achieved that the image data can be displayed and processed on external devices. In a further advantageous embodiment of the refrigeration device, the interface is a wireless interface. As a result, the technical advantage is achieved, for example, that the transmission is performed without a cabling effort.

In a further advantageous embodiment of the refrigeration device, the camera module comprises a compression device for compressing the image data. This will

For example, the technical advantage that reduces the amount of data on the data bus.

In a further advantageous embodiment of the refrigeration device is the

Compression device is designed to convert the image data into a JPEG format. As a result, for example, the technical advantage is achieved that a particularly efficient conversion is used.

In a further advantageous embodiment of the refrigeration device, the camera module comprises a microcontroller. As a result, for example, the technical advantage is achieved that the camera module can be easily integrated into the refrigeration device.

In a further advantageous embodiment of the refrigerator implements the

Data bus a USB standard, a l2C standard, an SPI standard, or a CAN standard. As a result, for example, the technical advantage is achieved that especially reliable and appropriate standards are used for transmission on the data bus.

In a further advantageous embodiment of the refrigeration device, the camera module is designed to be supplied with electrical power via the data bus. As a result, for example, the technical advantage is achieved that reduces cabling costs for the refrigeration device.

In a further advantageous embodiment of the refrigeration device, the refrigeration device comprises a controllable via the data bus illumination device. As a result, for example, the technical advantage is achieved that the refrigerated goods can be suitably illuminated during a recording.

In a further advantageous embodiment of the refrigeration device, the refrigeration device comprises a local distribution node for distributing data and control signals between the processor device and the camera modules. As a result, the technical advantage is achieved, for example, that increases the data throughput.

In a further advantageous embodiment of the refrigeration device, the refrigeration device comprises a connection for electrically connecting the camera module to the data bus. As a result, for example, the technical advantage is achieved that the camera module can be decoupled with the data bus and thereby can be easily replaced.

Embodiments of the invention are illustrated in the drawings and will be described in more detail below.

Show it:

Fig. 1 is a schematic view of a refrigerator; 2A is a perspective view of a refrigeration compartment with two camera modules.

Fig. 2B is a front view of the refrigerating compartment;

3A is a perspective view of a refrigeration compartment with a camera module; Fig. 3B is a plan view of the refrigerating compartment;

4 shows a perspective view of a refrigeration compartment with a plurality of camera modules;

5 shows two views of different camera modules;

Fig. 6 is a plan view and a side view of the refrigerating compartment in conjunction with a

Field Of View;

Fig. 7 is a plan view of the refrigeration compartment when using the multiple

Camera modules;

8 shows a perspective view of a refrigeration compartment with a sensor mat;

9 is a view of a refrigerator with a data bus.

10 shows a further view of a refrigerator with a data bus;

10A is a schematic view of an arrangement of the camera modules in a door;

Fig. 1 1 is a schematic view of a data bus with distribution nodes;

Fig. 12 is a view of a processor device;

Fig. 13 images of refrigerated goods in the refrigeration compartment and an image analysis;

14 is a view of a camera module operating device with a

touch-sensitive display;

Fig. 15 is a schematic view of a refrigerator with a camera field in one

Door; and

16A-C are schematic views of different connections of camera modules. Fig. 1 shows a refrigerator representative of a refrigerator 100 with an upper and lower refrigerator door 1 1 1. The refrigerator 100 is a stand-alone device (stand-alone device).

The refrigerator is used for example for cooling food and includes a refrigerant circuit with an evaporator, a compressor, a condenser and a throttle body. The evaporator is a heat exchanger in which after expansion the liquid refrigerant is absorbed by heat absorption from the medium to be cooled, i. the air inside the refrigerator is evaporated. The compressor is a mechanically operated component, the refrigerant vapor from the

Evaporator sucks and ejects at a higher pressure to the condenser. Of the

Condenser is a heat exchanger in which, after compression, the vaporized refrigerant is released by heat release to an external cooling medium, i. the ambient air is liquefied. The throttle body is a device for the continuous reduction of the pressure by cross-sectional constriction.

The refrigerant is a fluid used for heat transfer in the cryogenic system which absorbs heat at low temperatures and low pressure of the fluid and releases heat at higher temperature and pressure of the fluid, usually including changes in state of the fluid.

The refrigerator 100 includes the upper and lower door 1 1 1, which closes a refrigeration compartment inside the refrigerator 100. In addition, the refrigeration device 100 includes a side wall 109. Inside the refrigeration device 100 is an electronic camera module, which is capable of refrigerated goods in the interior of the refrigeration device 100 by means of a single image

to record and generate image data. The camera module 101 may, for example, be adjustable, pivotable or adjustable by the customer.

FIG. 2A shows a perspective view of the refrigeration compartment 105 with two camera modules 101. The refrigeration compartment 105 forms a receiving space for the refrigerated goods 103 in the interior of the refrigerator

Refrigeration device 100. Inside the refrigeration compartment 105 are as refrigerated goods 103rd

For example, a cup and a food bag. The two camera modules 101 are arranged on a front side 131 of the refrigeration compartment 105 at the same height and at a horizontal distance from each other, for example in the door 1 1 1 of the refrigerator 100. The refrigerated goods 103 is from the two camera modules 101 from the front of the refrigerator 100 ago added. The camera modules 101 on the front side 131 of the refrigeration compartment 105 enable the emulation of a user's view of the contents of the refrigeration compartment 105.

To illuminate the cold compartment 105 during a recording is a

Lighting device 102 is provided. The illumination device 102 is

For example, an infrared or a lighting with visible light. The lighting device 102 is arranged, for example, in a cover of the refrigeration compartment 105. One and the same picture can be taken with different lighting for additional picture information or a higher resolution. The design of the interior of the refrigeration compartment 105 may be optimized with respect to the camera module 105, for example with regard to adjustability, a shelf arrangement, the arrangement of the door racks to the shelves or an adjustability of

Door racks. The camera modules 101 are inclined, for example, to the center of the refrigerator. The camera modules 101 may be movable. A camera module 101 in a side wall has the advantage that a view of the door 1 1 1 is obtained. In addition, reflective coatings, glass beads, mini-retroreflectors can be used.

Freshness compartment does not have its own camera, transparent lid, transparent shelf, no opaque strip on the shelf edge

The camera modules 101 include a light sensor array with color sensors, such as a CCD chip. The light sensor field has, for example, a resolution of at least 640 × 480 pixels (VGA) and three color channels with a color depth of at least 8 bits per color channel, for example RGB color channels. The camera module 101 may, for example, be designed in such a way that it does not only record image data in the visible range but also additionally captures image data in the infrared range.

Usable CCD image sensors for the light sensor array consist of an array of photosensitive photodiodes. These can be rectangular, square or polygonal, with edge lengths of 1, 4 μηη to over 20 μηη. The larger the area of the photodiodes, the higher the photosensitivity and the dynamic range of the CCD image sensor. With the same sensor size, however, the image resolution is smaller in this case. The camera modules 101 have an interface for communication with a processor device. The obtained sensor data of the light sensor field can be processed with the processor device. The system has power only on request of the order of magnitude of 3 W and standby of 0, 1 W.

Fig. 2B shows a frontal view of the refrigeration compartment 105 from the perspective of one of

Camera modules 101. In the front view, the contours of the refrigerated goods 103 can be seen against the background of a rear side 129 of the refrigeration compartment 105.

By using two camera modules 101 which are arranged at a horizontal distance from each other, for example, two images of the goods to be cooled 103 can be received, which convey a three-dimensional image of the goods to be cooled 103 to a user in a stereoscopic view. In addition, when using two

Camera modules 101 the advantage that, even if one of the camera modules 101 by

Refrigerated goods 103 should be adjusted, the interior of the refrigeration compartment 105 can be received by the other of the camera modules 101.

Fig. 3A shows another perspective view of the refrigerating compartment 105 with a

Camera module 101. The camera module 101 is arranged in the compartment cover 1 13 of the refrigeration compartment 105. The arrangement of the camera module 101 in the compartment ceiling 1 13 is a fundamental change in the direction of view and a view of the contents of the

Refrigeration compartment 105 allows. 3B shows a plan view of the refrigeration compartment 105 from the perspective of the camera module 103 arranged in the compartment cover 13. This supervision can be used for the visualization of the contents of the refrigeration compartment 105.

In general, the camera module can also be arranged on the rear side 129 or a side wall 109 of the refrigeration compartment 105. This allows the field of vision for the

Visualization of the content can be expanded and different perspectives within the refrigeration compartment 105 or a panoramic view are included.

4 shows a perspective view of a refrigeration compartment 105 with a plurality of camera modules 101. The camera modules 101 are arranged such that the refrigerated goods 103 can be received from each side and at different viewing angles. By assembling the images taken by the individual camera modules 101, it is possible to generate a 360 ° view of the refrigerated goods 103 in the refrigeration compartment 105. So that the refrigerated goods 103 can be picked up by the camera module 101 with sufficient brightness, a lighting device 102 is realized in the interior of the refrigerating compartment 105. The camera module 101 and the illumination device 102 can be arranged, for example, in a common installation space. For example, a flash unit (flash light) for emitting a flash of light during a recording is arranged as illumination device 102 in the interior of the refrigeration device 100, so that a recording is obtained by means of flash light. The flash device is arranged, for example, on the camera module 101. The flash unit may be activated by an additional controller which outputs a control signal for each camera module 101. The light intensity of the flash can be measured with a special dual-function flash meter or light meter.

In addition, the refrigerator 100, a light-emitting diode array (LED array) as

Lighting device 102 include, which is arranged such that the refrigerated goods 103 is illuminated in the refrigeration compartment 105 as evenly as possible from all sides. The LED field can be controlled centrally. In addition, a homogeneous or adaptive illumination of the refrigeration compartment 105 can be realized by the light-emitting diode array in a simple and energy-saving manner, and a shadow cast can be reduced. In general, however, all other suitable lighting devices may be used, such as LED strips, incandescent or neon tubes. When the refrigerator door is open, room lighting can be used as interior lighting. Fig. 5 shows two different camera modules 101-1 and 101-2. For image acquisition, two types of camera modules 101 -1 and 101 -2 are available, which are suitable for installation in the refrigeration device 100. The finished camera modules 101-1 and 101-2 comprise, for example, a light sensor field, an objective and a microcontroller 104. Camera modules 101-1 and 101-2 with integrated microcontroller 104 can be integrated into the refrigeration device 100 in a simple manner. Does a driver exist for the respective one

Camera module 101-1 and 101-2, the integration is also simplified. if the

Camera modules 101 -1 and 101 -2 include a USB interface, the

Camera modules 101 -1 and 101 -2 directly to a USB data bus or a PC be connected. The camera modules 101-1 and 101-2 have an imager that provides raw image data for evaluation.

The camera modules 101-1 and 101-2 are protected against fruit acid, dust and moisture. The camera modules 101-1 and 101-2 can be arranged in an encapsulated housing or and with a lens heating by an IR LED and / or

Lens coating be provided to prevent fogging.

The first camera module 101 - 1 comprises a pinhole lens 135 and a board 133, on which a chip with a light sensor field and a microcontroller 104 are arranged. The camera module 101-1 is replaced by a pinhole camera 107 (pin hole).

Camera), which may include a glass cover against contamination. Although the light intensity of the pinhole camera 107 is smaller than that of focusing cameras, the depth of field of the pinhole camera 107 is larger. However, the lower light intensity can be easily compensated by a longer exposure time, since no movements take place inside the refrigeration compartment 105.

The second camera module 101-2 comprises an objective 109 and a printed circuit board 133, on which a chip with a light sensor field and a microcontroller are arranged. The objective 109 may, for example, have a focal length of less than 10 mm (miniature objective), so that the objective 109 can also generate a high depth of focus and a near-field can be imaged. The objective 109 is, for example, a wide-angle lens with a

Image opening angle of over 60 °. The image aperture angle describes the angular range that can be detected by the camera module 101 -2. In addition, the camera modules 101-1 and 101-2 may include other redirecting means such as foldable or rotating prisms, mirrors, lenses, or semiconductor mirrors.

The board 133 has, for example, dimensions of 32 mm x 32 mm. On the board 133, for example, electrical circuits for the microcontroller 104 and an interface for connecting the camera module 101 to a data bus are formed, so that image data can be provided as a file. The board 133 is formed for example in the door 1 1 1 in the foam.

The height of the camera module 101-1 and 101 -2 is from the lens 109 or the

Aperture 135 defined and is for example between 10 mm and 30 mm. Aperture diaphragms 135 allow a smaller depth of the camera module 101 of less than 20 mm. moreover For example, a camera module 101 -1 with a pinhole 135 requires a smaller opening or recess for mounting in a wall of the refrigeration device 100. For a camera module 101 -1 with a pinhole 135, the depth of the slot is, for example, 2 mm to 3 mm, whereas for a camera module 101-2 with a lens 109 the depth of

Recess is about 10 mm. The camera module 101 -1 with the aperture plate 135 can also be realized without a circuit board 133 and can then have dimensions of, for example, 10 mm × 10 mm × 7 mm. A pinhole camera 107 with a pinhole 135 can

for example, allow an image opening angle of 120 °.

The camera modules 101-1 and 101-2 may have different interfaces for transmitting image data, such as an RS232 interface, an SPI interface, an IIC interface or a USB interface.

6 shows a plan view and a side view of the refrigeration compartment 105 in conjunction with different image opening angles for a refrigeration compartment 105 with 20 cm × 60 cm × 50 cm. The image aperture angle describes the angular range that can be imaged by the camera module 101. At different image opening angles, different viewing areas of the camera module 101 result. Side of the camera module 101 are in the refrigeration compartment 105 individual areas 137, which are not visible and with larger

As the image opening angle becomes smaller.

Fig. 7 shows a plan view of the refrigeration compartment 105 when using the multiple

Camera Modules 105. On the left is shown an arrangement with four camera modules 101 with an image aperture angle of 40 °. In contrast, an arrangement with three camera modules 101 and an image opening angle of 60 ° is shown on the right. For larger

Image aperture angles may be less camera modules 101 used to cover a same field of view 139.

The image aperture angle determines the number of camera modules 101 to produce a closed image. Between the image opening angle and the number of

Camera modules 101 can be found depending on the size of the refrigeration compartment 105 a compromise. However, a larger aperture angle results in more distortion.

For example, a pincushion distortion 141 or a barrel distortion 143 results. The type and severity of the distortion 141 and 143 can be identified and compensated for further image processing. For this a pattern can be used. Fig. 8 shows a refrigeration compartment 105 with a sensor mat 1 15 for detecting a change in a load of a shelf 1 17. The film-like sensor mat 1 15 can

Detection of weight changes can be used. The sensor mat 1 15 is a few millimeters thick and is placed on the shelf 1 17. The sensor mat 1 15 measures the change of loads on the horizontal, flat and solid shelf 1 17 and is suitable for monitoring freestanding and stacked refrigerated goods 103 on the shelf 1 17. This allows, for example, levels of containers by comparing the weight with a Determine initial value. In addition, it is also possible to track repositioning of the item to be cooled 103, so that a detection of the position takes place.

In addition, the position of a newly set refrigerated goods 103-2 can be detected. If the refrigerated goods 103-2 are added to the refrigerated goods 103-1 on the shelf 1 17, for example, the sensor mat 1 17 recognizes the surface area on which the refrigerated goods 103-2 have been added. As a result, changes in the refrigeration compartment 105 can be detected in addition to an optical change in the image data acquired by the camera module 101.

In addition, the refrigeration device 100 may comprise a RFID tag reader, so that the refrigerated goods 103 can be detected in the refrigeration compartment 105 by means of a readout of the labels. The RFID tag reader is an electronic circuit capable of reading the information stored in electronic tags by means of radio waves.

Both the RFID tag reader and the sensor mat 15 can be provided as additional sensors for supplementing an optical image capture, for example as a replacement or supplement to the optical recognition of an EAN code. This increases reliability and allows more applications.

In addition, the camera module 101 may comprise a controllable pivoting device for rotating a recording direction of the camera module 101 and / or a controllable zoom device for enlarging a recording of the camera module 101. In addition, the camera module 101 may comprise a displacement device for the controllable horizontal and / or vertical displacement of the camera module 101 on a wall of the refrigeration device 100. The pivoting device, the displacement device and the zoom device can by means of a suitable mechanism in conjunction with an electromotive Drive be realized and controlled by a processor device via a data bus.

9 shows a refrigeration appliance 100 with the data bus 121. The refrigeration compartment 105 comprises four camera modules 101, which are arranged in the respective corners of the refrigeration compartment 105.

In addition, the refrigeration device 100 includes a processor device 1 19, to which the images are transmitted from the individual camera modules 101. The processor device 1 19 comprises a processor and a memory and serves to process the transmitted image data. A data bus 121 is used to transmit the image data from the camera module 101 to the processor device 1 19. A data bus is a system for data transmission between multiple participants over a common transmission path, in which the participants are not involved in the data transmission between other participants. Each of the four camera modules 101 is connected to a separate data bus 121. To this data buses 121 more devices can be connected and controlled. The data bus 121 is partially integrated in the side wall 109 of the refrigeration device 100. The data bus 121 is formed by electrical lines which transmit signals from the camera modules 101 to the processor device 1 19.

The data bus 121 is, for example, a serial bus with which the individual bits or data packets are transmitted in a serial data transmission successively (serially) via the line. The data bus 121 implements, for example, a USB standard, an I2C standard, an SPI standard or a CAN standard, or, for example, a DBus2 standard or DBus3 standard. The data bus 121 is for example a Hi-Speed Bus (USB). A serial data bus 121 has the advantage that it can be realized with a small number of lines.

The camera modules 101 and the processor device 1 19 also have a serial interface for connecting the data bus 121. The use of camera modules 101 and systems for which drivers already exist is advantageous. The data bus 121 may be pre-terminated for this purpose with connection devices, such as connectors, for connecting the camera modules 101. In this case, in case of a defect, the camera module 101 can be easily replaced and replaced. In addition, the data bus 121 and connected components can be configured in this way be that the components, such as camera modules 101 or

Lighting devices 102 are supplied via the data bus 121 with electrical power or electricity. This reduces a wiring effort inside the

Refrigerating appliance 100. FIG. 10 shows a further refrigerating appliance 100 with the data bus 121. The refrigeration compartment 105 includes four camera modules 101 arranged in the respective corners of the refrigeration compartment 105. Each two of the four camera modules 101 are on the same data bus 121

connected. Two of the camera modules 101 are arranged in a door 1 1 1 of the refrigeration device 100 and generate a frontal view of the refrigeration compartment 105. The data bus 121 to these camera modules 101 is guided by a cable passage through a hinge 123 of the door 1 1 1.

The respective camera modules 101 are assigned a logical or physical address via which the camera modules 101 can be addressed individually on the data bus 121. In addition, additional local distribution nodes can be provided as needed. The duration of an image request from the processor device 1 19 via the data bus 121 should not exceed a predetermined period of time.

10A shows a schematic view and a cross-sectional view of an arrangement of six camera modules 101 on the inside of a door 1 1 1. On the outside, the door 1 1 1 comprises a door handle 169. The camera modules 101 capture the image data from a same area inside a refrigeration compartment 105 from different angles ago. Two each of the camera modules 101 form a pair with a first and a second

Camera module 101 for a stereoscopic image acquisition.

Each two of the camera modules 101 are arranged in the door 1 1 1 as a pair at the same height. A first camera module 101 of the pair is arranged on a right-hand side rail 167 and a second camera module 101 is arranged on a left-hand side rail 167. The camera modules 101 are arranged in a corner of the door pillars 167 on the inside of the door 1 1 1. The camera modules 101 are integrated, for example, in a body of the door 1 1 1, so that they do not protrude from the inside of the door 1 1 1.

Each camera module 101 is associated with a lighting device 102, for example in the form of a white light-emitting diode in the door 1 1 1. The respective Lighting device 102 is arranged obliquely next to the respective camera module 101.

In the middle of the door 1 1 1 a door rack 165 is formed, can be placed in the refrigerated goods 103, such as bottles. The door rack 165 is not height adjustable. Two of the camera modules 101 are located directly below the door compartment 165, so that they can not be adjusted by refrigerated goods 103 inside the refrigerator 100. Two further camera modules 101 are arranged in an upper region of the door 1 1 1, so that these can not be adjusted by refrigerated goods and an image taken from obliquely above can be performed.

The respective pairs of camera modules 101 are arranged at different heights on the inside of the door 1 1 1, so that the recording of different

Viewpoints can be gained. For this purpose, for example, each pair of camera modules 101 detects a same area inside a refrigeration compartment 105 from different angles.

The door 1 1 1 comprises an integrated data bus 121 for connecting the camera modules 101 to a processor device 1 19, so that the image data obtained by the camera modules 101 can be transmitted for further processing. For example, all six camera modules 101 are connected to the same data bus 121. This data bus 121 may be formed as a serial data bus, so that the transmission of the image data can be performed with a low cabling overhead. In each case, the first and the second camera module 101 of a pair can each comprise a zoom device for synchronously enlarging the stereoscopic image acquisition. In general, the camera modules 101 may also be arranged at other locations inside the refrigeration device, such as at a rear side of the refrigeration device 100.

1 1 shows a schematic view of a data bus 121 in a refrigeration device 100 with a plurality of distribution nodes 125. The distribution nodes 125 each form an auxiliary module for connecting the camera modules 101 and have corresponding interfaces. The

Distribution modules 125 are implemented, for example, by means of an embedded platform, such as a beagleboard.

The distribution nodes 125 serve to distribute image data and control signals between the processor device 1 19 and the camera modules 101 and can Form image conversion units. In addition, in the distribution node 125 a

Data compression can be performed. In addition, a protocol translation can take place in the distribution node 125.

The camera modules 101 connected to the data bus 121 can have a

Microcontroller, by which a compression of the acquired image data takes place. Thereby, a compression means for compressing the image data can be realized. In this compression, the image data obtained in the bitmap format, for example, converted into image data in the JPEG format and transmitted in the JPEG format over the data bus 121. This has the advantage that the transmission is performed with less data and takes a shorter time span.

FIG. 12 shows a perspective view of the processor device 1 19. The

Processor device 1 19 includes an interface 145 for transmitting the image data to an external network. The interface 145 is for example a wireless WLAN interface, a Bluetooth interface, an Ethernet interface or a

Mobile radio interface such as GSM, UMTS, or LTE. Via the interface 145, the image data can be provided to an external network. Optionally, a direct connection to a personal computer is possible by means of the interface 145. The personal computer can be used to process the sensor data, such as removing distortion, removing noise, sharpening artificial sharpening, or increasing contrast.

The processor device 19 serves to process the sensor and image data from the camera modules 101. For image recognition, a program library with algorithms for image processing and machine vision can be used, such as, for example

OpenVC. Realizable methods of image recognition are, for example, the detection of a change, the recognition of a barcode or the recognition of defined logos

(Pattern recognition). As the operating system of the processor device 1 19 Linux can be used.

By means of the processor device 19, the image data can be retrieved from the camera modules 101 and the image data thus transmitted can be processed. In this

In connection with this, image processing can be carried out, for example a

Suppress or combine image data into larger units. In addition, an analysis and evaluation of the image data can be made, for example, recognize of difference images or difference objects. In this evaluation, refrigerated goods 103 received by the camera modules 101 can be identified and associated

Determine additional information. The image data can be provided via a wireless network. In addition, the processor device 1 19 can control a

Illuminator 102 are performed to illuminate the refrigeration compartment 105 during a recording. For this purpose, the refrigeration device 100 comprises a

Lighting device 102 which is connected to the data bus 121 and is controllable via this.

13 shows a first image taken by the camera module 101 from first image data 147 and a second image taken from the camera module 101 from second image data 149. The second image data 149 has been recorded at a later time than the first image data 147. By means of the processor device 1 19 or an external camera module operating device on which an application (app) is running, a

Frischebestimmungseinrichtung be realized, which is used to determine a

Freshness of the chilled goods based on the first image data 147 and the second

Image data 149 is used. The time interval between the first and second image data 147 and 149 is one day, for example.

In both cases, the lighting device 102 is largely the same

Lighting conditions in the refrigeration compartment 105 realized to ensure comparability of time-shifted recordings. So that the image data 147 and 149 are not falsified by ambient light, which enters the refrigeration compartment 105 when the door 1 1 1 of the refrigerator 100 opens, the refrigeration appliance 100 comprises a sensor for detecting an opening of the door 1 1 1. Is by the sensor detects that the door 1 1 1 of the refrigeration device 100 is open, no acquisition of the image data is performed. Only when the door is closed 1 1 1, the image data are detected. The image data are taken after closing the door 1 1 1, after a uniform illumination of the interior of the refrigeration compartment 105 has been made.

In addition, by placing the first image data and the second image data in comparison, it is possible to ascertain or remove refrigerated goods 103. The image data 151 shows the subtraction of the second image data 149 from the first image data 147. The first image data 147 and the second image data 149 are, for example, three-color image data (RGB) having a color resolution of 24 bits per pixel.

By the subtraction, the difference and the change between both image data 147 and 149 can be detected. In the example shown, the image area 153 represents the refrigerated goods 103, which has been removed from the refrigeration compartment 105. A fade in or

Hiding the contents of the refrigeration compartment 105 and the determined change can be made on a display of the refrigeration device 100, for example a tablet PC or a smartphone. In addition, a contour of the goods to be cooled 103 and the effected change can be highlighted.

Similarly, a temporal change of the refrigerated goods 103 can be determined, which consists of a perishable goods. One formed by the processor device 1 19 or the camera module operating device by means of an application

Fresh determination device, determines the freshness state of the chilled goods 103 based on a color difference between the first image data 147 and the second image data 149. Based on a color difference can then be determined with a suitable algorithm, whether the refrigerated goods 103 is still fresh or durable. In addition, the data on changes in weight that are detected by the sensor mat 15 can be used for freshness determination.

Moreover, it is possible that the freshness-determining device is designed to detect the type of goods to be cooled 103 on the basis of the first image data or the second image data. The freshness determination device can automatically detect, for example, what kind of perishable goods the refrigerated goods 103 are. For example, the freshness determination device is able to detect by image analysis whether the goods to be cooled 103 in the refrigeration compartment 105 consist of bananas or tomatoes. One

Image evaluation algorithm compares the recording of the camera module 101 with pre-stored image data and thereby detects the refrigerated goods 103. Should an automatic detection of the type of refrigerated goods 103 fail, this can

Refrigeration device 100 include an input device for manually entering the type of refrigerated goods 103. This is formed for example by a touch-sensitive display or a display device, are displayed on the different types of goods to be refrigerated 103 in menu form. A user selects by tapping on the appropriate menu item from a specific item to be refrigerated, this has deposited in the refrigeration compartment 105. Based on entered or determined type of refrigerated goods 103, the freshness state of the refrigerated goods 103 can be determined more precisely by means of a color difference.

For detecting a freshness state, a special camera module 101 can be used, which determines the image data not only in the visible spectral range but at the same time also in the infrared spectral range. The camera module 101 may include a dichroic element that separates a beam path of the infrared light from that of the visible light.

The dichroic element is, for example, an optical prism that splits a light beam into two beams of different spectra or colors. The dichroic element is made, for example, from a glass in which certain surfaces are provided with dichroic mirrors which reflect or transmit light depending on its wavelength.

The visible light is detected by a first light sensor field and the separated infrared light by a second light sensor field. The number, arrangement and type of

Camera modules 101 and the lighting is generally determined depending on the desired execution concept.

In order to illuminate the refrigerated goods 103 during the image recording, a lighting device 102 is provided in the refrigeration compartment 105, which illuminates the refrigeration compartment 105 in the same way at each shot. To a spectral analysis of the goods to be cooled 103 at a

For example, the illumination device 102 can emit monochromatic and narrow-band light of a predetermined frequency. For example, the illumination device 102 comprises a tunable light emitting diode for emitting a monochromatic light of variable wavelength. However, the illumination device 102 may also be designed such that it emits polychromatic white light or infrared light.

The refrigeration device 100 may include a control and display device that displays the specific freshness state of the refrigerated goods 103 to a user. Exceeds the detected

Color difference or a color change of, for example, fruit or vegetables a predetermined limit, a user can be notified via the display device. As a control and display device, a tablet PC or smartphone can be used. If the image data 147 and 149 are present in the processor device 1 19 or the display device, further information can be extracted therefrom by means of image recognition algorithms, for example by means of an application on the camera module operating device. This information can then be displayed on the camera module operator panel.

For example, it is not only possible to detect changes on the basis of the recorded image data 147 and 149, but it is also possible to detect bar codes (bar codes) or QR codes of the goods to be cooled 103 or to perform a character recognition. A further evaluation of the codes can take place via a product database on the Internet, in which the corresponding codes are stored. In addition, the determined product data can be assigned to recorded changes in the image data.

In addition, image analysis using pattern recognition enables identification of product logos. For this purpose, the product logos should be clearly recognizable and not be covered by other refrigerated goods 103. Unless the inclusion sufficient details of

Including product logos, a reliable visual recognition of the product can be guaranteed. The processor device 1 19 can be trained by means of suitable algorithms on a variety of products.

In combination with the product recognition, a list of contents of the refrigeration device 100 can be created. The product recognition is realized by means of an optical recognition of an EAN code on the product. Products without an EAN code, such as fruit or vegetables, can be selected manually from a selection list. This can be a

Product registration by recognizing the EAN code. After the product has been placed in the refrigeration device 100, the resulting change in the image data is then assigned and highlighted (tracing).

The product database of the refrigeration device 100 can be extended to the entire kitchen area. For this purpose, a storage location is first selected and a

Product registration performed by detecting the EAN code. Subsequently, the filing of the product is carried out by the user. The inventory can be updated when goods are removed. Product recognition based on the EAN code and assignment to the image data are also possible.

An indication of additional information, such as a minimum shelf life (MHD) or a minimum stock quantity, for a registered product can be used for

Monitoring can be used. In this case, the user can the additional information (MHD) when it selects the product on a display or in the database. If the stock of the product is running out or the product runs out, the user can be notified via the camera module operating unit.

In this context, for example, a fusion of the image data with the

Product data by storing the contour information from an image recognition after the registration of a product instead. By means of the product database further applications can be realized in addition to product monitoring. For example, these applications can be in the process of creating a recipe proposal based on the inventory list or creating a shopping list for frequently sourced products. An adaptive system can take into account the habits of the user. An online interface to the inventory or image data provides convenient on-the-go access. In addition to pure image recognition and image display, the product survey and product database with the display of additional information for the user is an added value.

14 shows a view of a portable camera module operating device 300 with a touch-sensitive display 303 (touchscreen). The camera module operating device 300 is, for example, a tablet PC which is connected to the refrigeration device 100 by means of a WLAN or a smartphone which is connected to the refrigeration device 100 by means of the mobile radio network. However, the camera module operating device 300 can also be connected to the refrigeration device 100 by means of a USB interface. However, the USB interface can also be used for charging the camera module operating device 300.

On the camera module operating device 300, an application or application can be executed, such as a computer program. For mounting the

Camera module operating device 300 may be provided an assembly kit. The

Camera module operator 300 is connected to a router via a wireless network. The camera module operating device 300 can act as an image server or as a graphics computer and save the image data via a WLAN. The portable camera module operator 300 may be replaced or updated by a user.

Between the portable camera module operating device 300 and the refrigeration device 100, data can be exchanged bidirectionally. The wireless camera module operating device 300 is part of a refrigeration system and serves to display the image data captured by the camera modules 101. The user can thus display the displayed refrigeration compartment 105 in the Select the application on the camera module operating device 300. In addition, a

Change view by swiping on the touch-sensitive display 303 up and down or by tilting the camera module operating device 300 to the right or left. For example, an angular attitude of the camera module operating device 300 can determine which refrigeration compartment 105 is displayed.

The camera modules 101 inside the refrigeration compartment 105 have a large

Image opening angle. For a closed view of the front view, multiple camera modules 101 are used. The refrigeration device 100 may comprise a plurality of camera modules 101, which are arranged in the refrigeration compartment 105 in such a way that the refrigerated goods 103 are picked up from all sides and an all-round view is produced by assembling the individual images. The selection of the viewing angle of the panoramic view of the goods to be refrigerated 103 is carried out by wiping on the touch-sensitive display 303 to the left and right.

A camera module 101 with a large image opening angle (fish eye), which is arranged in a ceiling 1 13, allows a view of the refrigerated goods 103 in the refrigeration compartment 105. Another camera module 101 is disposed in a door 1 1 1 of the refrigerator 100. The refrigeration compartment 105 additionally comprises a lighting device 102 for uniform illumination of the refrigerated goods 103. In this case and in all other cases, the

Selection of the respective view by tilting the camera module operating device 300 done.

In general, the refrigeration device 100 as part of the refrigeration device system 200 may include a first camera module 101 for acquiring first image data of the refrigerated goods 103 from a first perspective and a second camera module 101 for acquiring second image data of the first

Refrigerated goods 103 include from a second perspective. A change of perspective may generally occur upon touching the touch-sensitive display 303. The refrigeration device 100 may include any illumination device 102 that is used for

Illuminate the refrigerated goods 103 in the refrigeration compartment 105 during capture of image data is suitable. The illumination device 102 may be controllable by the camera module operating device 300.

On the side of the camera module operating device 300, a menu selection 305 is displayed, via which the respective refrigeration compartment 105 can be selected with the corresponding camera module 101. Upon selecting the cold compartment 105 by touching the touch-sensitive display 303, its content is displayed. In addition, the respective view through Wipe on the touch-sensitive display 103 are selected. The view can be enlarged or reduced, for example by a

Moving apart with the fingers on the touch-sensitive display 103. In addition, additional information can be displayed, such as a

Fresh state of the chilled goods 103 or changes in the image data compared to stored earlier image data.

The camera module operating device 300 can compare the acquired image data of the refrigerated goods 103 with stored image data of the refrigerated goods 103. In addition, that can

Camera module operating device 300 identify the refrigerated goods 103 on the basis of the captured image data by means of an analysis algorithm. The user can select the image data of the camera module 101 and the processed images on the camera module operating device 300, view and display additional information, such as the assignment of a bar code to a specific product. The image data and the associated

Additional information is requested, processed and provided by the processor device 1 19 of the refrigeration device 100. For this, the camera data and

Additional information from an external central computer system queried, processed and provided. An association between the refrigeration device 100 and the

Camera module operating device 300 can take place via a central processing unit,

for example via Home Connect.

15 shows a schematic view of a refrigeration device 100 with a camera field 127 in the door 1 1 1. The refrigeration device 100 includes a camera field 127 with a plurality of camera-like arranged camera modules 101, which are arranged on the inside of the door 1 1 1. Each of these camera modules 101 generates a single image in a frontal view from the inside of the refrigeration device 100. The camera modules 101 of the camera field 127 are arranged such that from the individual images a composite view of the complete front view can be generated as with the door 1 1 1 open. From the camera field 127 in the refrigerator door 1 1 1 a panoramic image is composed, which represents a closed shot of the front. To put together the

To improve frames, a calibration of the camera modules 101 can be performed.

Furthermore, the refrigeration device 100 comprises a position detection device for detecting a position of the device 100 placed on the outside of the refrigeration device

Camera module operating device 300. This position detection means allow For example, determine a coordinate pair that indicates at which point of the front of the refrigerator 100, the camera module operating device 300 is placed.

Following this, an image area assigned to the coordinate pair can be transmitted from the previously assembled view to the camera module operating device 300. This creates the impression that a user is using the wireless

Camera module operating device 300 x-ray through the wall of the refrigerator 100 can look through. The user can also enlarge areas of the composite view. During a movement of the camera module operating device 300 in front of the front side of the refrigeration device 100, the corresponding image area shifts on the

Camera module operating device 300 and the position corresponding to the position of the camera module operating device 300 cold compartment 105 is displayed. While the camera module operating device 300 travels outside via the refrigeration device 100, the displayed image section follows the position of the camera module operating device 300. Even if only one camera module is used, an image section can be shifted. Inside the refrigerator 100 here is a uniform illumination of all areas instead.

The complexity of software development is generally related to the desired additional functions, such as image capture, image recognition, product recognition (EAN), recognition of changes in an image, a product database, and an association of the products in the image data by recognition ,

In the refrigeration device 100, only the camera modules 101 and an interface (interface) to the camera module operating device 300 may be provided. The control of the camera modules 101 and the evaluation of the image data via the camera module operating device 300. Therefore, in the refrigeration device 100 a minimal control electronics can be realized. A user is not bound to outdated hardware, but can replace the camera module operating device 300 as desired. Optionally, a door opening and closing signal may be communicated, on the basis of which the camera module operating device 300 controls the camera modules 101. The camera module operating device 300 can transmit the image data via WLAN and display the content of the refrigeration device 100.

For example, the image data is obtained at each door closing, then provided or transferred to an external storage location and then transmitted immediately upon request. Illumination remains immediately after a door closing for a period of 300 - 500 ms available. This has the advantage that a last changed current content is detected, a small additional heat input takes place immediately after a large heat input and lighting is only possible through a normal interior lighting. 16A to 16C show different connections of camera modules 101. The

Integration of camera modules 101 and the controller can be done in various ways.

In Fig. 16A, the camera module 101 has a control 155 and a camera 157 as an integrated unit and is connected to a network 161. In Fig. 16B, a plurality of camera modules 101 with the controls 155 are integrated

Image processing unit or a distribution node 125 connected to the network 161. In Fig. 16C, a multi-camera module is connected to the network 161. The multi-camera module may be connected to the network 161 with an integrated or separate image processing unit 125. The multi-control 155 serves to control a plurality of cameras 157 of the multi-camera module. In addition, an interface to a GSM network 163 may be provided. The camera 157, the controls 155, the multi-controller 155, and the image processing unit 125 may be physically identical units that perform different tasks.

The control of the camera modules 101 can be done by a computer in the refrigerator or be provided separately from the refrigeration appliance control, for example on the

Camera module operating device 300, such as a smartphone and mobile tablet PC. The

Camera module operating device 300 with the screen can be attached to the door 1 1 1.

The camera module operator 300 may execute a program or application. The used for the evaluation of the image data and control of the camera modules 101. For example, the application is used for freshness determination based on the image data obtained. The freshness of the chilled goods 103 can be distinguished from a maturity of the chilled goods 103.

For products that "freeze", the cell structure is affected by the effects of cold, such as bananas, tomatoes, cucumbers, zucchini, aubergines, tropical fruits. in the

In general, weight, smell, taste, color, texture, such as furrows, size, diameter or volume can be seen. By means of the texture can be observed whether a product is still crisp. A Turgeszenz indicates how strongly the cells are filled with water and is a measure of the freshness.

A freshness detection application captures, evaluates, and provides information about products to other processing facilities. For many products, a base date is the best before date (MHD), which is associated with the product and provided externally.

Unpackaged foods are not associated with the product

Best before date. In addition, an alternative expiration date can be used, which depends on the product, the storage time, a refrigerator setting, a temperature stratification of the specific refrigeration device 100 and a humidity. In this context, a traffic light system is possible:

Red: non-recordable best before date

Yellow: alternative expiration date

Green: best before date

Refrigerated goods such as meat may be opaque packed so that the condition of the meat is unobservable. However, the packaging may be labeled and have a barcode and a best-before date. For some products, there is an increased health hazard beyond the best before date, such as minced meat, poultry meat or fish products. For these products, the traffic light can be adjusted. To assess whether a product is still usable, the product specification contains a value allowing a loss-type-specific assessment.

Life color texture

medium relative to absolute weight size

e.g. red / brown

Apple + + + +

Salad + - + (-)

Melon + + - + The relative color judgment may be judged by a difference from the first image and a shading threshold. The maturity detection is based on the

Color shading.

The application may display instructions for storage in the refrigeration device 100, such as

"This food should not be stored in the refrigerator" "Minced meat on the top floor! Note: Store down / in a colder zone" or

"Note: Setting the refrigerator to colder or adjusting the automatic function will reduce the cost of storage and overcompensate electricity costs." Some products have a Time Temperature Integrator (TTI) that indicates product-specific activation

Color change labels are sensory readable. Through a coupling with a

Warehouse Management System (Storage Guide) can be used to generate a nutritional recommendation. When filling the refrigeration device 100, an inventory list of all stored objects can be created by dragging them past a camera module.

In addition, the application can play the long-term change in fast motion. During a virtual tour, the application can display an image of the refrigeration appliance content on the camera module operating device 300. The functions can be a combination of language and barcode (EAN / QR), image recognition and image representation. Products can be added. It can be synced with a photo and shopping lists can be generated.

In addition, the refrigeration device 100 can be configured such that a user can request image data from a mobile telephone, so that even at remote locations a possible fresh observation is possible. For this a GSM camera with MMS dispatch can be used. The connection between camera and mobile phone takes place via the GSM infrastructure. By means of an SMS, an image can be requested, which is then transmitted as an MMS image. Photos and videos can also be sent by e-mail or MMS with a time and date stamp. A photo or video delivery can be made on demand at regular intervals. The photo resolution of the MMS is for example 160x120, 320x240, 640x480 pixels. A recording takes place in the internal memory, if no GSM signal is present. Power is supplied by battery, power adapter or USB port. The control can be carried out directly on the refrigeration unit 100, on a personal computer or by SMS from the mobile phone.

By means of the mobile phone, a convenient remote control via Android App can be enabled. The GSM camera can be activated via GSM via call or SMS and supports SMS commands. In addition, a power failure message can be issued. The angle of view of the camera is 60 ° or 90 ° with attachable magnetic wide-angle lens, In addition, a flash function can be implemented.

In addition, a UMTS / HSDPA camera (WCDMA TD-SCDMA) with a photo resolution of 1.3 megapixels can be used. The UMTS / HSDPA camera has a tilt and pan function and a night vision function. A UMTS video server is located separately from the camera.

In addition, a RJ45-10 / 100M Ethernet interface can be provided with the without WLAN. Network-capable camera module operating devices 300 are, for example, a laptop or a television. This can be a separation between multimedia components and

Refrigeration technology can be achieved. A multimedia interface may be a standard multimedia interface. In addition, glass fibers can be used with a lens at one end. Several glass fibers can be read out with the same color sensor.

All features explained and shown in connection with individual embodiments of the invention may be provided in different combinations in the article according to the invention, in order to simultaneously realize their advantageous effects. The scope of the present invention is given by the claims and is not limited by the features illustrated in the specification or shown in the figures. LIST OF REFERENCE NUMBERS

100 refrigeration unit

101 camera module

101 -1 camera module

101 -2 camera module

102 lighting device

103 refrigerated goods

104 microcontroller

105 refrigeration compartment

107 pinhole camera

109 side wall

1 1 1 door

1 13 ceiling

1 15 sensor mat

1 17 shelf

1 19 processor device

121 data bus

123 hinge

125 distribution node

127 camera field

129 back side

131 front side

133 board

135 pinhole

137 invisible area

139 viewing area

141 distortion

143 distortion

145 interface

147 image data

49 image data

151 subtraction

153 image area

155 control

157 camera 161 network

163 GSM network

165 supporters

167 door spar

169 door handle

200 refrigeration system

300 Indicator / Camera Module Operator 303 Touch Indicator 305 Menu selection

Claims

Characterized in that the refrigeration device (100) comprises a camera module (101) for detecting image data of a refrigerated goods (103) in a cold compartment (105) of the refrigeration device (100), a processor device (1 19) for processing the Image data and a data bus (121) for transmitting the image data from the camera module (101) to the processor device (1 19).

2. Refrigerating appliance (100) according to claim 1, characterized in that the data bus (121) is a serial data bus.

3. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the data bus (121) by a hinge (123) in a door (1 1 1) of the refrigeration device (100) is guided.

4. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the refrigeration device (100) comprises a plurality of camera modules (101) which are connected to the data bus (121).

5. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the camera module (101) is associated with a unique address via which the

Camera module (101) by means of the data bus (121) is controlled.

6. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the processor device (1 19) comprises an interface (145) for transmitting the image data to an external network.

7. Refrigerating appliance (100) according to claim 6, characterized in that the interface (145) is a wireless interface.

8. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the camera module (101) comprises a compression device for compressing the image data.

9. Refrigerating appliance (100) according to claim 8, characterized in that the

Compression device is designed to convert the image data into a JPEG format.

10. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the camera module (101) comprises a microcontroller (104).

1 1. Refrigeration appliance (100) according to one of the preceding claims, characterized in that the data bus (121) implements a USB standard, a l ² C standard, an SPI standard, or a CAN standard.

12. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the camera module (101) is designed to be supplied via the data bus (121) with electrical power.

13. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the refrigeration device (100) via the data bus (121) controllable

Lighting device (102).

14. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the refrigeration device (100) comprises a local distribution node (125) for distributing data and control signals between the processor device (1 19) and the camera modules (101).

15. Refrigerating appliance (100) according to one of the preceding claims, characterized in that the refrigeration device (100) has a connection for the electrical connection of the

Camera module (101) to the data bus (121).