JP2020188474A - Shovel - Google Patents

Abstract

To allow an operator of a shovel to intuitively grasp a position of an object present around the shovel.SOLUTION: A shovel 60 includes: three cameras 2 mounted at three locations, a left part, a right part, and a rear part of an upper swing body 63 so as to image three directions of a left side, a right side, and a rear side of the upper swing body 63; a cab 64 mounted on the upper swing body 63; and a display unit 5 installed in the cab 64. Then, the shovel 60 causes the display unit 5 to simultaneously display a first image corresponding to at least one of three directions and a second image of the upper swing body 63 viewed from above, generated by composing images of the three directions, simultaneously displaying at least the left, right, and rear regions of the upper swing body 63a regardless of an orientation of a lower swing body 61.SELECTED DRAWING: Figure 21

Description

The present invention relates to excavators.

Conventionally, there is known a peripheral monitoring device that emits an alarm sound when an operator is detected within the monitoring range of an obstacle detector mounted on an excavator (see, for example, Patent Document 1). In addition, whether or not the worker who has entered the work area set around the excavator is a collaborative worker is judged from the light emission pattern of the LED attached to the helmet of the worker and whether or not an alarm sound is output. An alarm system is known to determine whether or not (see, for example, Patent Document 2). Further, there is known a safety device that communicates between a forklift and an operator who works in the vicinity (around) of the forklift and controls the output of an alarm sound based on this communication (see, for example, Patent Document 3). ).

Japanese Unexamined Patent Publication No. 2008-179940 JP-A-2009-193494 JP-A-2007-310587

However, in all the techniques of Patent Documents 1 to 3, the alarm sound is emitted from the same buzzer or speaker regardless of the direction in which the worker who has entered the predetermined range is viewed from the worker such as the excavator. Output. Therefore, an operator such as a shovel cannot intuitively grasp in which direction the operator is present even if he / she hears the alarm sound.

In view of the above points, it is an object of the present invention to enable the operator of the excavator to intuitively grasp the position of an object existing around the excavator.

In order to achieve the above object, the excavator according to the embodiment of the present invention is attached to the lower traveling body, the upper rotating body rotatably mounted on the lower traveling body, and the upper rotating body, and is attached to the attachment. The left and right parts of the upper swing body so as to image the included boom, the arm attached to the boom and included in the attachment, and the left, right, and rear three directions of the upper swing body. , And three image pickup devices mounted at three locations at the rear, an cab mounted on the upper swing body, and a display unit installed in the cab, and the display unit is provided with the three directions. Regardless of the orientation of the lower traveling body, which is generated by synthesizing the first image corresponding to at least one of the directions and the images in the three directions, at least to the left and right of the upper rotating body. , And a second image of the upper swivel body looking down from the sky, which simultaneously displays the rear area, and is displayed at the same time.

By the above-mentioned means, it is possible to provide a shovel that enables the operator of the shovel to intuitively grasp the position of an object existing around the shovel.

It is a block diagram which shows schematic structure example of the image generation apparatus which concerns on embodiment of this invention. It is a figure which shows the configuration example of the excavator which mounts an image generator. It is a figure which shows an example of the spatial model which the input image is projected. It is a figure which shows an example of the relationship between a spatial model and an image plane to be processed. It is a figure for demonstrating the correspondence between the coordinates on the input image plane, and the coordinates on a space model. It is a figure for demonstrating the correspondence between coordinates by the coordinate correspondence means. It is a figure for demonstrating the action of a group of parallel lines. It is a figure for demonstrating the action of the auxiliary line group. It is a flowchart which shows the flow of the processing target image generation processing and output image generation processing. This is a display example of the output image (No. 1). It is a top view (No. 1) of the excavator on which the image generator is mounted. It is a figure (the 1) which shows the input image of each of three cameras mounted on a shovel, and the output image generated by using the input image. It is a figure for demonstrating the image disappearance prevention processing which prevents the disappearance of an object in the overlapping part of the imaging space of each of two cameras. It is a comparison diagram which shows the difference between the output image shown in FIG. 12 and the output image obtained by applying the image disappearance prevention processing to the output image of FIG. It is a figure (No. 2) which shows the input image of each of three cameras mounted on a shovel, and the output image generated by using the input image. It is a correspondence table (No. 1) which shows the correspondence relationship between the determination result of the person existence determination means, and the input image used for generating an output image. This is a display example of the output image (No. 2). It is a top view (No. 2) of the excavator on which the image generator is mounted. It is a correspondence table (No. 2) which shows the correspondence relationship between the determination result of the presence / absence determination means, and the input image used for generating an output image. This is a display example of the output image (No. 3). This is a display example of the output image (No. 4). It is a flowchart which shows the flow of an alarm control process. It is a figure which shows an example of the transition of the output image displayed in the alarm control processing. It is a block diagram which shows the other configuration example of the peripheral monitoring apparatus schematicly.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram schematically showing a configuration example of an image generation device 100 according to an embodiment of the present invention.

The image generation device 100 is an example of a peripheral monitoring device for a work machine that monitors the periphery of the work machine, and includes a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, a person detection sensor 6, and a human detection sensor 6. It is composed of an alarm output unit 7. Specifically, the image generation device 100 generates an output image based on the input image captured by the camera 2 mounted on the work machine, and presents the output image to the operator. Further, the image generation device 100 switches the content of the output image to be presented based on the output of the person detection sensor 6.

FIG. 2 is a diagram showing a configuration example of the excavator 60 as a work machine on which the image generator 100 is mounted. The excavator 60 is placed on a crawler type lower traveling body 61 via a swivel mechanism 62. The swivel body 63 is mounted so as to be swivelable around the swivel shaft PV.

Further, the upper swivel body 63 is provided with a cab (driver's cab) 64 on the front left side thereof, an excavation attachment E on the front center portion thereof, and a camera 2 (right side camera 2R, rear camera 2B) on the right side surface and the rear surface thereof. ) And a person detection sensor 6 (right side person detection sensor 6R, rear person detection sensor 6B). The display unit 5 is installed at a position in the cab 64 that is easily visible to the operator. Further, in the cab 64, an alarm output unit 7 (right side alarm output unit 7R, rear alarm output unit 7B) is installed on the right inner wall and the rear inner wall.

Next, each component of the image generation device 100 will be described.

The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), and the like. In this embodiment, the control unit 1 stores, for example, a program corresponding to each of the coordinate mapping means 10, the image generation means 11, the presence / absence determination means 12, and the alarm control means 13 described later in the ROM or NVRAM. The CPU is made to execute the processing corresponding to each means while using the RAM as the temporary storage area.

The camera 2 is a device for acquiring an input image projecting the surroundings of the excavator 60. In this embodiment, the camera 2 is, for example, a right side camera 2R and a rear camera 2B attached to the right side surface and the rear surface of the upper swivel body 63 so that the area of the operator's blind spot in the cab 64 can be imaged (FIG. See 2.). Further, the camera 2 includes an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 2 may be attached to a position other than the right side surface and the rear surface (for example, the front surface and the left side surface) of the upper swing body 63, and a wide-angle lens or a fisheye lens is attached so as to be able to image a wide range. You may be.

Further, the camera 2 acquires an input image according to the control signal from the control unit 1, and outputs the acquired input image to the control unit 1. When the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the camera 2 sends a corrected input image corrected for apparent distortion and tilt caused by using those lenses to the control unit 1. Output. Further, the camera 2 may directly output the input image without correcting the apparent distortion and tilt to the control unit 1. In that case, the control unit 1 corrects the apparent distortion and tilt.

The input unit 3 is a device for allowing the operator to input various information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

The storage unit 4 is a device for storing various types of information, such as a hard disk, an optical disk, or a semiconductor memory.

The display unit 5 is a device for displaying image information, for example, a liquid crystal display or a projector installed in the cab 64 (see FIG. 2) of the excavator 60, and various types output by the control unit 1. Display the image.

The human detection sensor 6 is a device for detecting a person existing around the excavator 60. In this embodiment, the person detection sensor 6 is attached to, for example, the right side surface and the rear surface of the upper swing body 63 so as to be able to detect a person existing in the area of the operator's blind spot in the cab 64 (see FIG. 2). ..

The human detection sensor 6 is a sensor that distinguishes and detects a person from an object other than the human, and is, for example, a sensor that detects an energy change in the corresponding monitoring space, and is a charcoal infrared sensor, a bolometer infrared sensor, or the like. Includes a motion detection sensor that uses an output signal from an infrared camera or the like. In this embodiment, the person detection sensor 6 uses a pyroelectric infrared sensor, and detects a moving object (moving heat source) as a person. Further, the monitoring space of the right-side person detection sensor 6R is included in the imaging space of the right-side camera, and the monitoring space of the rear person detection sensor 6B is included in the imaging space of the rear camera 2B.

Similar to the camera 2, the person detection sensor 6 may be attached to a position other than the right side surface and the rear surface of the upper swivel body 63 (for example, the front surface and the left side surface), and the front surface of the upper swivel body 63. It may be attached to any one of the left side surface, the right side surface, and the rear surface, or may be attached to all surfaces.

The alarm output unit 7 is a device that outputs an alarm to the operator of the excavator 60. For example, the alarm output unit 7 is an alarm device that outputs at least one of sound and light, and includes an audio output device such as a buzzer and a speaker, and a light emitting device such as an LED and a flashlight. In this embodiment, the alarm output unit 7 is a buzzer that outputs an alarm sound, and is a right side alarm output unit 7R attached to the right inner wall of the cab 64 and a rear alarm output unit attached to the rear inner wall of the cab 64. It is composed of 7B (see FIG. 2).

Further, the image generation device 100 generates a processing target image based on the input image, and performs image conversion processing on the processing target image so that the positional relationship with surrounding objects and a sense of distance can be intuitively grasped. After generating the output image to be used, the output image may be presented to the operator.

The “processed image” is an image that is generated based on the input image and is the target of image conversion processing (for example, scale conversion processing, affine transformation processing, distortion conversion processing, viewpoint conversion processing, and the like). Specifically, the "processed image" is, for example, an input image taken by a camera that captures the ground surface from above, and includes an image in the horizontal direction (for example, an empty portion) due to its wide angle of view. It is an image suitable for image conversion processing generated from. More specifically, the input image is projected onto a predetermined spatial model so that the horizontal image is not displayed unnaturally (for example, the empty part is not treated as if it is on the ground surface), and then the image is projected. It is generated by reprojecting a projected image projected onto a spatial model onto another two-dimensional plane. The image to be processed may be used as it is as an output image without performing image conversion processing.

The "spatial model" is the projection target of the input image. Specifically, the "spatial model" is composed of at least one or a plurality of planes or curved surfaces including a plane or a curved surface other than the processing target image plane which is the plane on which the processing target image is located. The plane or curved surface other than the processing target image plane, which is the plane on which the processing target image is located, is, for example, a plane parallel to the processing target image plane or a plane or curved surface forming an angle with the processing target image plane. ..

The image generation device 100 may generate an output image by performing an image conversion process on the projected image projected on the spatial model without generating the image to be processed. Further, the projected image may be used as it is as an output image without performing image conversion processing.

FIG. 3 is a diagram showing an example of the spatial model MD on which the input image is projected, and FIG. 3 left is a diagram showing the relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from the side. The right figure of FIG. 3 shows the relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from above.

As shown in FIG. 3, the spatial model MD has a semi-cylindrical shape, and has a plane region R1 inside the bottom surface thereof and a curved surface region R2 inside the side surface thereof.

Further, FIG. 4 is a diagram showing an example of the relationship between the spatial model MD and the processing target image plane, and the processing target image plane R3 is, for example, a plane including the plane region R1 of the spatial model MD. In FIG. 4, for clarification, the space model MD is shown in a cylindrical shape instead of the semi-cylindrical shape as shown in FIG. 3, but the space model MD is either a semi-cylindrical shape or a cylindrical shape. It may be. The same applies to the following figures. Further, the image plane R3 to be processed may be a circular region including the plane region R1 of the space model MD or an annular region not including the plane region R1 of the space model MD as described above.

Next, various means included in the control unit 1 will be described.

The coordinate mapping means 10 is a means for associating the coordinates on the input image plane on which the input image captured by the camera 2 is located, the coordinates on the spatial model MD, and the coordinates on the processing target image plane R3. In the present embodiment, the coordinate mapping means 10 includes, for example, various parameters related to the camera 2 preset or input via the input unit 3, and a predetermined input image plane, spatial model MD, and the like. And, based on the mutual positional relationship of the processing target image plane R3, the coordinates on the input image plane, the coordinates on the spatial model MD, and the coordinates on the processing target image plane R3 are associated with each other. The various parameters related to the camera 2 are, for example, the optical center of the camera 2, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method, and the like. Then, the coordinate mapping means 10 stores the correspondence between them in the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 of the storage unit 4.

When the coordinate mapping means 10 does not generate the processing target image, the coordinate mapping between the coordinates on the spatial model MD and the coordinates on the processing target image plane R3, and the spatial model / processing target of the corresponding relationship. The storage in the image correspondence map 41 is omitted.

The image generation means 11 is a means for generating an output image. In this embodiment, the image generation means 11 performs scale conversion, affine transformation, or distortion transformation on the image to be processed, so that the coordinates on the image plane R3 to be processed and the output image are located on the output image plane. Associate with coordinates. Then, the image generation means 11 stores the correspondence relationship in the processing target image / output image correspondence map 42 of the storage unit 4. Then, the image generation means 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41. Generate an output image. The value of each pixel is, for example, a luminance value, a hue value, a saturation value, or the like.

Further, the image generation means 11 is an output image plane in which the coordinates on the processing target image plane R3 and the output image are located based on various parameters related to the virtual camera that are set in advance or input via the input unit 3. Associate with the above coordinates. The various parameters related to the virtual camera are, for example, the optical center of the virtual camera, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method, and the like. Then, the image generation means 11 stores the correspondence relationship in the processing target image / output image correspondence map 42 of the storage unit 4. Then, the image generation means 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41. Generate an output image.

Note that the image generation means 11 may generate an output image by changing the scale of the image to be processed without using the concept of a virtual camera.

Further, when the image generation means 11 does not generate the image to be processed, the image generation means 11 associates the coordinates on the spatial model MD with the coordinates on the output image plane according to the image conversion process performed. Then, the image generation means 11 generates an output image by associating the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / spatial model correspondence map 40. In this case, the image generation means 11 omits associating the coordinates on the processing target image plane R3 with the coordinates on the output image plane and storing the correspondence in the processing target image / output image correspondence map 42. ..

Further, the image generation means 11 switches the content of the output image based on the determination result of the presence / absence determination means 12. Specifically, the image generation means 11 switches the input image used for generating the output image based on the determination result of the presence / absence determination means 12, for example. The details of the switching of the input image used for generating the output image and the output image generated based on the switched input image will be described later.

The presence / absence determination means 12 is a means for determining the presence / absence of a person in each of the plurality of monitoring spaces set around the work machine. In this embodiment, the presence / absence determination means 12 determines the presence / absence of people around the excavator 60 based on the output of the person detection sensor 6.

Further, the presence / absence determination means 12 may determine the presence / absence of a person in each of the plurality of monitoring spaces set around the work machine based on the input image captured by the camera 2. Specifically, the presence / absence determination means 12 may determine the presence / absence of people around the work machine by using image processing techniques such as optical flow and pattern matching. The presence / absence determination means 12 may determine the presence / absence of people around the work machine based on the output of an image sensor different from the camera 2.

Alternatively, the presence / absence determination means 12 determines the presence / absence of a person in each of the plurality of monitoring spaces set around the work machine based on the output of the person detection sensor 6 and the output of the image sensor such as the camera 2. May be good.

The alarm control means 13 is a means for controlling the alarm output unit 7. In this embodiment, the alarm control means 13 controls the alarm output unit 7 based on the determination result of the presence / absence determination means 12. The details of the control of the alarm output unit 7 by the alarm control means 13 will be described later.

Next, an example of specific processing by the coordinate mapping means 10 and the image generation means 11 will be described.

The coordinate associating means 10 can associate the coordinates on the input image plane with the coordinates on the spatial model by using, for example, Hamilton's quaternion.

FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the spatial model. The input image plane of the camera 2 is represented as one plane in the UVW Cartesian coordinate system with the optical center C of the camera 2 as the origin. The spatial model is represented as a three-dimensional plane in the XYZ Cartesian coordinate system.

First, the coordinate mapping means 10 translates the origin of the XYZ coordinate system to the optical center C (origin of the UVW coordinate system), and then shifts the X axis to the U axis, the Y axis to the V axis, and the Z axis. Rotate the XYZ coordinate system so that each coincides with the -W axis. This is to convert the coordinates on the spatial model (coordinates on the XYZ coordinate system) into the coordinates on the input image plane (coordinates on the UVW coordinate system). The symbol "-" of "-W axis" means that the direction of the Z axis and the direction of the -W axis are opposite to each other. This is because the UVW coordinate system has the front of the camera in the + W direction, and the XYZ coordinate system has the vertical lower direction in the −Z direction.

When there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinate system. Therefore, the coordinate mapping means 10 translates the XYZ coordinate system with respect to each of the plurality of UVW coordinate systems. Rotate.

In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 becomes the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the −W axis, and the X axis is U. It is realized by rotating it so that it matches the axis. Therefore, the coordinate mapping means 10 can combine these two rotations into one rotation operation by describing this conversion in Hamilton's quaternion.

By the way, the rotation for matching one vector A with another vector B corresponds to the process of rotating by the angle formed by the vector A and the vector B about the normal of the surface stretched by the vector A and the vector B. .. Then, assuming that the angle is θ, the angle θ is calculated from the inner product of the vector A and the vector B.

It is represented by.

Further, the unit vector N of the normal of the surface extending between the vector A and the vector B is obtained from the outer product of the vector A and the vector B.

Will be represented by.

The quaternion is when i, j, and k are imaginary units, respectively.

It is a hypercomplex number that satisfies, and in this embodiment, the quaternion Q has a real component as t and a pure imaginary component as a, b, and c.

The conjugate quaternion of the quaternion Q is represented by

It is represented by.

The quaternion Q can express a three-dimensional vector (a, b, c) with pure imaginary components a, b, and c while setting the real component t to 0 (zero), and t, a, b. It is also possible to express a rotation motion about an arbitrary vector by each component of, c.

Further, the quaternion Q can be expressed as one rotation operation by integrating a plurality of consecutive rotation operations. Specifically, the quaternion Q is, for example, when an arbitrary point S (sx, sy, sz) is rotated by an angle θ with an arbitrary unit vector C (l, m, n) as an axis. The point D (ex, ey, ez) can be expressed as follows.

Here, in this embodiment, assuming that the quaternion representing the rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X', so that the point X'is

It is represented by.

Further, in this embodiment, assuming that the quaternion representing the rotation that makes the line connecting the point X'on the X axis and the origin coincide with the U axis is Qx, "the Z axis matches the -W axis, and further. , The quaternion R representing "rotation that makes the X-axis coincide with the U-axis"

It is represented by.

From the above, the coordinates P'when the arbitrary coordinates P on the spatial model (XYZ coordinate system) are expressed by the coordinates on the input image plane (UVW coordinate system) are

It is represented by. Further, since the quaternary number R is invariant in each of the cameras 2, the coordinate mapping means 10 thereafter simply executes this calculation to input the coordinates on the spatial model (XYZ coordinate system) in the image plane (UVW). Coordinate system) It can be converted to the above coordinates.

After converting the coordinates on the spatial model (XYZ coordinate system) into the coordinates on the input image plane (UVW coordinate system), the coordinate mapping means 10 is formed by the line segment CP'and the optical axis G of the camera 2. Calculate the incident angle α. The line segment CP'is a line segment connecting the optical center C (coordinates on the UVW coordinate system) of the camera 2 and the coordinate P'representing an arbitrary coordinate P on the spatial model in the UVW coordinate system.

Further, the coordinate mapping means 10 calculates the declination φ and the length of the line segment EP'in the plane H parallel to the input image plane R4 (for example, the CCD plane) of the camera 2 and including the coordinates P'. The line segment EP'is a line segment connecting the intersection E of the plane H and the optical axis G and the coordinates P', and the deviation angle φ is formed by the U'axis and the line segment EP'on the plane H. The angle to do.

In the optical system of a camera, the image height h is usually a function of the incident angle α and the focal length f. Therefore, the coordinate mapping means 10 has a normal projection (h = ftanα), a normal projection (h = fsinα), a stereographic projection (h = 2ftan (α / 2)), and an equal stereographic projection (h = 2fsin (α/2)). )), Equal distance projection (h = fα) and other appropriate projection methods are selected to calculate the image height h.

After that, the coordinate mapping means 10 decomposes the calculated image height h into U component and V component on the UV coordinate system by the declination φ, and uses a numerical value corresponding to the pixel size per pixel of the input image plane R4. Divide. As a result, the coordinate mapping means 10 can associate the coordinates P (P') on the spatial model MD with the coordinates on the input image plane R4.

Assuming that the pixel size per pixel in the U-axis direction of the input image plane R4 is a _U and the pixel size per pixel in the V-axis direction of the input image plane R4 is a _V , the coordinates P on the spatial model MD The coordinates (u, v) on the input image plane R4 corresponding to (P') are

It is represented by.

In this way, the coordinate mapping means 10 associates the coordinates on the spatial model MD with the coordinates on one or more input image planes R4 existing for each camera, and the coordinates on the spatial model MD and the camera identifier. , And the coordinates on the input image plane R4 are associated and stored in the input image / spatial model correspondence map 40.

Further, since the coordinate mapping means 10 calculates the coordinate conversion using quaternions, it has an advantage that gimbal lock is not generated unlike the case where the coordinate conversion is calculated using Euler angles. .. However, the coordinate mapping means 10 is not limited to the one that calculates the coordinate conversion using the quaternion, and may calculate the coordinate conversion using the Euler angles.

When it is possible to associate the coordinates on the plurality of input image planes R4, the coordinate mapping means 10 inputs the coordinates P (P') on the spatial model MD with respect to the camera having the smallest incident angle α. It may be associated with the coordinates on the image plane R4, or may be associated with the coordinates on the input image plane R4 selected by the operator.

Next, among the coordinates on the spatial model MD, the process of reprojecting the coordinates on the curved surface region R2 (coordinates having a component in the Z-axis direction) onto the processing target image plane R3 on the XY plane will be described.

FIG. 6 is a diagram for explaining the correspondence between the coordinates by the coordinate matching means 10. F6A is a diagram showing the correspondence between the coordinates on the input image plane R4 of the camera 2 that employs normal projection (h = ftanα) as an example and the coordinates on the spatial model MD. The coordinate mapping means 10 is set so that each of the line segments connecting the coordinates on the input image plane R4 of the camera 2 and the coordinates on the spatial model MD corresponding to the coordinates passes through the optical center C of the camera 2. Associate both coordinates.

In the example of F6A, the coordinate mapping means 10 associates the coordinates K1 on the input image plane R4 of the camera 2 with the coordinates L1 on the plane region R1 of the spatial model MD, and the coordinates K2 on the input image plane R4 of the camera 2. Corresponds to the coordinates L2 on the curved surface region R2 of the spatial model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

When the camera 2 employs a projection method other than the normal projection (for example, normal projection, stereographic projection, equal-angle projection, equal-distance projection, etc.), the coordinate mapping means 10 is used for each projection method. Correspondingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the spatial model MD.

Specifically, the coordinate mapping means 10 has a predetermined function (for example, normal projection (h = fsinα), stereographic projection (h = 2ftan (α / 2)), equal-angle projection (h = 2fsin (α / 2)). 2)), equidistant projection (h = fα), etc.), the coordinates on the input image plane are associated with the coordinates on the spatial model MD. In this case, the line segments K1-L1 and the line segments K2-L2 do not pass through the optical center C of the camera 2.

F6B is a diagram showing the correspondence between the coordinates on the curved surface region R2 of the spatial model MD and the coordinates on the processing target image plane R3. The coordinate mapping means 10 introduces a parallel line group PL located on the XZ plane and forming an angle β with the image plane R3 to be processed. Then, in the coordinate mapping means 10, the coordinates on the curved surface region R2 of the spatial model MD and the coordinates on the processing target image plane R3 corresponding to the coordinates are both placed on one of the parallel line group PL. , Associate both coordinates.

In the example of F6B, the coordinate associating means 10 associates the coordinates L2 on the curved surface region R2 of the spatial model MD with the coordinates M2 on the processing target image plane R3 on a common parallel line.

The coordinate mapping means 10 can also map the coordinates on the plane region R1 of the spatial model MD to the coordinates on the processing target image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved surface region R2. is there. However, in the example of F6B, the plane region R1 and the processing target image plane R3 are common planes. Therefore, the coordinates L1 on the plane region R1 of the spatial model MD and the coordinates M1 on the processing target image plane R3 have the same coordinate values.

In this way, the coordinate mapping means 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3. It is stored in the spatial model / processing target image correspondence map 41.

FIG. F6C is a diagram showing a correspondence relationship between the coordinates on the image plane R3 to be processed and the coordinates on the output image plane R5 of the virtual camera 2V that employs normal projection (h = ftanα) as an example. The image generation means 11 so that each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates pass through the optical center CV of the virtual camera 2V. And associate both coordinates.

In the example of F6C, the image generation means 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (plane region R1 of the spatial model MD), and the virtual camera 2V The coordinates N2 on the output image plane R5 are associated with the coordinates M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

When the virtual camera 2V employs a projection method other than the normal projection (for example, normal projection, stereographic projection, equal-angle projection, equal-distance projection, etc.), the image generation means 11 uses each projection method. Correspondingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

Specifically, the image generation means 11 has a predetermined function (for example, normal projection (h = fsinα), stereographic projection (h = 2ftan (α / 2)), equidistant angle projection (h = 2fsin (α / 2)). )), Equidistant projection (h = fα), etc.), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated with each other. In this case, the line segments M1-N1 and the line segments M2-N2 do not pass through the optical center CV of the virtual camera 2V.

In this way, the image generation means 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and obtains the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3. It is associated and stored in the processing target image / output image correspondence map 42. Then, the image generation means 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41. Generate an output image.

Note that F6D is a diagram in which F6A to F6C are combined, and shows the mutual positional relationship between the camera 2, the virtual camera 2V, the plane region R1 and the curved surface region R2 of the spatial model MD, and the image plane R3 to be processed.

Next, with reference to FIG. 7, the operation of the parallel line group PL introduced by the coordinate mapping means 10 for associating the coordinates on the spatial model MD with the coordinates on the processing target image plane R3 will be described.

The left figure of FIG. 7 is a view when an angle β is formed between the parallel line group PL located on the XZ plane and the image plane R3 to be processed. On the other hand, the right figure of FIG. 7 is a diagram when an angle β1 (β1> β) is formed between the parallel line group PL located on the XZ plane and the image plane R3 to be processed. Further, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in the left figure of FIG. 7 and the right figure of FIG. 7 corresponds to the coordinates Ma to Md on the image plane R3 to be processed. Further, the respective intervals of the coordinates La to Ld in the left figure of FIG. 7 are equal to the respective intervals of the coordinates La to Ld in the right figure of FIG. 7. The parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from all points on the Z axis toward the image plane R3 to be processed. To do. The Z axis in this case is referred to as a "reprojection axis".

As shown in the left view of FIG. 7 and the right view of FIG. 7, the distance between the coordinates Ma to Md on the processing target image plane R3 increases the angle between the parallel line group PL and the processing target image plane R3. It decreases linearly with. That is, it decreases uniformly regardless of the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md. On the other hand, in the example of FIG. 7, the coordinate group on the plane region R1 of the spatial model MD is not converted to the coordinate group on the processing target image plane R3, so that the distance between the coordinate groups does not change. ..

As for the change in the spacing of these coordinate groups, only the image portion corresponding to the image projected on the curved surface region R2 of the spatial model MD among the image portions on the output image plane R5 (see FIG. 6) is linearly enlarged or expanded. It means that it will be reduced.

Next, an alternative example of the parallel line group PL introduced by the coordinate mapping means 10 for associating the coordinates on the spatial model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

FIG. 8 left is a diagram in which all of the auxiliary line groups AL located on the XZ plane extend from the start point T1 on the Z axis toward the processing target image plane R3. On the other hand, the right figure of FIG. 8 is a diagram in which all of the auxiliary line group AL extends from the start point T2 (T2> T1) on the Z axis toward the processing target image plane R3. Further, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in the left figure of FIG. 8 and the right figure of FIG. 8 corresponds to the coordinates Ma to Md on the image plane R3 to be processed. In the example shown on the left side of FIG. 8, the coordinates Mc and Md are not shown because they are outside the region of the image plane R3 to be processed. Further, the distance between the coordinates La to Ld in the left figure of FIG. 8 is equal to the distance between the coordinates La to Ld in the right figure of FIG. The auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from an arbitrary point on the Z axis toward the image plane R3 to be processed. To do. As in FIG. 7, the Z axis in this case is referred to as a “reprojection axis”.

As shown in the left figure of FIG. 8 and the right figure of FIG. 8, the distance between the coordinates Ma to Md on the image plane R3 to be processed is the distance (height) between the start point and the origin O of the auxiliary line group AL. Decreases non-linearly as increases. That is, the larger the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md, the larger the decrease width of each interval becomes. On the other hand, in the example of FIG. 8, the coordinate group on the plane region R1 of the spatial model MD is not converted to the coordinate group on the processing target image plane R3, so that the distance between the coordinate groups does not change. ..

The change in the spacing of these coordinate groups corresponds to the image projected on the curved surface region R2 of the spatial model MD in the image portion on the output image plane R5 (see FIG. 6) as in the case of the parallel line group PL. It means that only the image portion is non-linearly enlarged or reduced.

In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the spatial model MD, and the spatial model MD. The image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface region R2 of the above can be enlarged or reduced linearly or non-linearly. Therefore, the image generator 100 does not affect the road surface image in the vicinity of the excavator 60 (virtual image when the excavator 60 is viewed from directly above), and the object located around the excavator 60 (horizontal direction from the excavator 60). The object in the image when the surroundings are viewed) can be quickly and flexibly enlarged or reduced, and the visibility of the blind spot region of the excavator 60 can be improved.

Next, referring to FIG. 9, a process in which the image generation device 100 generates a processing target image (hereinafter, referred to as “processing target image generation processing”), and an output image using the generated processing target image. The process of generating (hereinafter, referred to as "output image generation process") will be described. Note that FIG. 9 is a flowchart showing the flow of the processing target image generation processing (steps S1 to S3) and the output image generation processing (steps S4 to S6). Further, the arrangement of the camera 2 (input image plane R4), the spatial model (plane area R1 and curved surface area R2), and the image plane R3 to be processed is determined in advance.

First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the spatial model MD by the coordinate mapping means 10 (step S1).

Specifically, the coordinate mapping means 10 acquires an angle formed between the parallel line group PL and the image plane R3 to be processed. Then, the coordinate mapping means 10 calculates a point where one of the parallel line group PLs extending from one coordinate on the processing target image plane R3 intersects the curved surface region R2 of the spatial model MD. Then, the coordinate mapping means 10 derives the coordinates on the curved surface region R2 corresponding to the calculated points as one coordinate on the curved surface region R2 corresponding to the one coordinate on the processing target image plane R3, and determines the correspondence relationship. It is stored in the spatial model / processing target image correspondence map 41. The angle formed between the parallel line group PL and the image plane R3 to be processed may be a value stored in advance in the storage unit 4 or the like, and the operator dynamically moves through the input unit 3. It may be a value to be entered.

Further, when one coordinate on the processing target image plane R3 matches one coordinate on the plane region R1 of the space model MD, the coordinate mapping means 10 converts the one coordinate on the plane region R1 into the processing target image. It is derived as one coordinate corresponding to that one coordinate on the plane R3, and the correspondence relationship is stored in the spatial model / processing target image correspondence map 41.

After that, the control unit 1 associates one coordinate on the spatial model MD derived by the above process with the coordinate on the input image plane R4 by the coordinate mapping means 10 (step S2).

Specifically, the coordinate associating means 10 acquires the coordinates of the optical center C of the camera 2 that normally employs projection (h = ftanα). Then, the coordinate mapping means 10 is a line segment extending from one coordinate on the spatial model MD, and calculates a point at which the line segment passing through the optical center C intersects the input image plane R4. Then, the coordinate mapping means 10 derives the coordinates on the input image plane R4 corresponding to the calculated points as one coordinate on the input image plane R4 corresponding to the one coordinate on the spatial model MD, and determines the correspondence relationship. It is stored in the input image / spatial model correspondence map 40.

After that, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the spatial model MD and the coordinates on the input image plane R4 (step S3). Then, when the control unit 1 determines that all the coordinates are not yet associated (NO in step S3), the control unit 1 repeats the processes of steps S1 and S2.

On the other hand, when it is determined that all the coordinates are associated with each other (YES in step S3), the control unit 1 ends the processing target image generation processing and then starts the output image generation processing. Then, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the output image plane R5 by the image generation means 11 (step S4).

Specifically, the image generation means 11 generates an output image by performing scale transformation, affine transformation, or distortion transformation on the image to be processed. Then, the image generation means 11 determines the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5, which are determined by the contents of the scale conversion, the affine transformation, or the distortion conversion applied to the processing target image. -Store in the output image correspondence map 42.

Alternatively, when the image generation means 11 generates an output image using the virtual camera 2V, the image generation means 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the projection method adopted. The correspondence may be stored in the processing target image / output image correspondence map 42.

Alternatively, when the image generation means 11 generates an output image using a virtual camera 2V that employs normal projection (h = ftanα), the image generation means 11 acquires the coordinates of the optical center CV of the virtual camera 2V. Then, the image generation means 11 calculates a line segment extending from one coordinate on the output image plane R5, and a point at which the line segment passing through the optical center CV intersects the processing target image plane R3. Then, the image generation means 11 derives the coordinates on the processing target image plane R3 corresponding to the calculated points as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5. In this way, the image generation means 11 may store the correspondence relationship in the processing target image / output image correspondence map 42.

After that, the image generation means 11 of the control unit 1 refers to the input image / spatial model correspondence map 40, the spatial model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42. Then, the image generation means 11 has a correspondence relationship between the coordinates on the input image plane R4 and the coordinates on the space model MD, a correspondence relationship between the coordinates on the space model MD and the coordinates on the processing target image plane R3, and the processing target. The correspondence between the coordinates on the image plane R3 and the coordinates on the output image plane R5 is traced. Then, the image generation means 11 acquires the values (for example, luminance value, hue value, saturation value, etc.) possessed by the coordinates on the input image plane R4 corresponding to each coordinate on the output image plane R5. The acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). When a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generation means 11 has each of the plurality of coordinates on the plurality of input image planes R4. A statistical value based on the value may be derived, and the statistical value may be adopted as the value of the one coordinate on the output image plane R5. The statistical value is, for example, an average value, a maximum value, a minimum value, an intermediate value, or the like.

After that, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the input image plane R4 (step S6). Then, when the control unit 1 determines that the values of all the coordinates are not yet associated (NO in step S6), the control unit 1 repeats the processes of steps S4 and S5.

On the other hand, when it is determined that the values of all the coordinates are associated with each other (YES in step S6), the control unit 1 generates an output image and ends this series of processing.

If the image generation device 100 does not generate the processing target image, the processing target image generation processing is omitted. In this case, the "coordinates on the processing target image plane" in step S4 in the output image generation processing are read as "coordinates on the spatial model".

With the above configuration, the image generation device 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the object around the excavator 60 and the excavator 60.

Further, the image generation device 100 executes coordinate mapping so as to trace back from the processing target image plane R3 to the input image plane R4 via the spatial model MD. As a result, the image generation device 100 can surely make each coordinate on the processing target image plane R3 correspond to one or a plurality of coordinates on the input image plane R4. Therefore, the image generation device 100 can quickly generate a higher quality image to be processed as compared with the case where the coordinates are associated in the order from the input image plane R4 to the image plane R3 to be processed via the spatial model MD. Can be done. When the coordinates are associated in the order from the input image plane R4 to the processing target image plane R3 via the spatial model MD, each coordinate on the input image plane R4 is converted into one or a plurality of coordinates on the processing target image plane R3. It is possible to surely correspond to the coordinates. However, some of the coordinates on the processing target image plane R3 may not be associated with any of the coordinates on the input image plane R4, and in that case, some of the coordinates on the processing target image plane R3. It is necessary to perform interpolation processing or the like.

Further, the image generator 100 changes the angle formed between the parallel line group PL and the image plane R3 to be processed when only the image corresponding to the curved surface region R2 of the spatial model MD is enlarged or reduced. By rewriting only the portion related to the curved surface region R2 in the spatial model / processing target image correspondence map 41, the desired enlargement or reduction can be realized without rewriting the contents of the input image / spatial model correspondence map 40. ..

Further, when changing the appearance of the output image, the image generation device 100 simply rewrites the processing target image / output image correspondence map 42 by changing the values of various parameters related to scale conversion, affine transformation, or distortion conversion. It is possible to generate a desired output image (scale conversion image, affine transformation image or distortion conversion image) without rewriting the contents of the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41.

Similarly, when the viewpoint of the output image is changed, the image generation device 100 simply changes the values of various parameters of the virtual camera 2V and rewrites the processing target image / output image correspondence map 42, and the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the spatial model / processing target image correspondence map 41.

FIG. 10 is a display example when the output image generated by using the input images of the two cameras 2 (right side camera 2R and rear camera 2B) mounted on the excavator 60 is displayed on the display unit 5. ..

The image generation device 100 generates an image to be processed by projecting the input images of the two cameras 2 onto the plane region R1 and the curved surface region R2 of the spatial model MD and then reprojecting them onto the processing target image plane R3. To do. Then, the image generation device 100 generates an output image by performing an image conversion process (for example, scale conversion, affine transformation, distortion conversion, viewpoint conversion processing, etc.) on the generated image to be processed. In this way, the image generation device 100 looks down on the vicinity of the excavator 60 from the sky (image in the plane region R1) and an image in which the surroundings are viewed in the horizontal direction from the excavator 60 (image in the image plane R3 to be processed). Generates an output image that displays and at the same time. Hereinafter, such an output image will be referred to as a peripheral monitoring virtual viewpoint image.

When the image generation device 100 does not generate the image to be processed, the virtual viewpoint image for peripheral monitoring is subjected to image conversion processing (for example, viewpoint conversion processing) on the image projected on the spatial model MD. Generated by.

Further, the virtual viewpoint image for peripheral monitoring is trimmed into a circle so that the image when the excavator 60 performs a turning operation can be displayed without discomfort, and the center CTR of the circle is on the cylindrical center axis of the spatial model MD, and It is generated so as to be on the swivel axis PV of the excavator 60. Therefore, the peripheral monitoring virtual viewpoint image is displayed so as to rotate about the central CTR according to the turning operation of the excavator 60. In this case, the central axis of the cylinder of the spatial model MD may or may not coincide with the reprojection axis.

The radius of the space model MD is, for example, 5 meters. Further, the angle formed by the parallel line group PL with the processing target image plane R3 or the starting point height of the auxiliary line group AL is the maximum reachable distance of the excavation attachment E (for example, 12 meters) from the turning center of the excavator 60. It is set so that when an object (for example, a worker) is present at a position separated by (for example, a worker), the object is displayed sufficiently large (for example, 7 mm or more) on the display unit 5. obtain.

Further, as the peripheral monitoring virtual viewpoint image, the CG image of the excavator 60 may be arranged so that the front of the excavator 60 coincides with the upper part of the screen of the display unit 5 and the turning center thereof coincides with the center CTR. .. This is to make it easier to understand the positional relationship between the excavator 60 and the object appearing in the output image. As the virtual viewpoint image for peripheral monitoring, a frame image including various information such as a direction may be arranged around the virtual viewpoint image.

Next, the details of the peripheral monitoring virtual viewpoint image generated by the image generation device 100 will be described with reference to FIGS. 11 to 14.

FIG. 11 is a top view of the excavator 60 equipped with the image generation device 100. In the embodiment shown in FIG. 11, the excavator 60 includes three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) and three person detection sensors 6 (left side person detection sensor 6L, right side). It is equipped with a person detection sensor 6R and a rear person detection sensor 6B). The regions CL, CR, and CB shown by the alternate long and short dash lines in FIG. 11 indicate the imaging spaces of the left side camera 2L, the right side camera 2R, and the rear camera 2B, respectively. Further, the regions ZL, ZR, and ZB shown by the dotted lines in FIG. 11 indicate the monitoring spaces of the left side person detection sensor 6L, the right side person detection sensor 6R, and the rear person detection sensor 6B, respectively. Further, the excavator 60 includes a display unit 5 and three alarm output units 7 (left side alarm output unit 7L, right side alarm output unit 7R, and rear alarm output unit 7B) in the cab 64.

In this embodiment, the monitoring space of the person detection sensor 6 is narrower than the imaging space of the camera 2, but the monitoring space of the person detection sensor 6 may be the same as the imaging space of the camera 2, and is larger than the imaging space of the camera 2. It may be wide. Further, the monitoring space of the human detection sensor 6 is located in the vicinity of the excavator 60 in the imaging space of the camera 2, but may be in a region farther from the excavator 60. Further, the monitoring space of the human detection sensor 6 has an overlapping portion in the portion where the imaging space of the camera 2 overlaps. For example, in the overlapping portion between the imaging space CR of the right side camera 2R and the imaging space CB of the rear camera 2B, the monitoring space ZR of the right side person detection sensor 6R overlaps with the monitoring space ZB of the rear person detection sensor 6B. However, the monitoring space of the human detection sensor 6 may be arranged so as not to cause duplication.

FIG. 12 is a diagram showing input images of each of the three cameras 2 mounted on the excavator 60 and output images generated by using the input images.

The image generation device 100 generates an image to be processed by projecting the input images of the three cameras 2 onto the plane area R1 and the curved surface area R2 of the spatial model MD and then reprojecting them onto the image plane R3 to be processed. .. Further, the image generation device 100 generates an output image by performing an image conversion process (for example, scale conversion, affine transformation, distortion conversion, viewpoint conversion processing, etc.) on the generated image to be processed. As a result, the image generator 100 obtains an image of the vicinity of the excavator 60 looking down from the sky (an image in the plane region R1) and an image of the surroundings in the horizontal direction from the excavator 60 (an image in the image plane R3 to be processed). Generates a virtual viewpoint image for peripheral monitoring to be displayed at the same time. The image displayed in the center of the peripheral monitoring virtual viewpoint image is the CG image 60CG of the excavator 60.

In FIG. 12, the input image of the right-side camera 2R and the input image of the rear camera 2B each capture a person in the overlapping portion between the imaging space of the right-side camera 2R and the imaging space of the rear camera 2B (right side). See the area R10 surrounded by the two-point chain line in the input image of the square camera 2R and the area R11 surrounded by the two-point chain line in the input image of the rear camera 2B).

However, assuming that the coordinates on the output image plane are associated with the coordinates on the input image plane for the camera with the smallest incident angle, the output image causes the person in the overlapping portion to disappear (one point in the output image). See region R12 surrounded by chain lines.)

Therefore, in the output image portion corresponding to the overlapping portion, the image generation device 100 associates the region on which the coordinates on the input image plane of the rear camera 2B are associated with the coordinates on the input image plane of the right camera 2R. The area is mixed to prevent the objects in the overlapping part from disappearing.

FIG. 13 is a diagram for explaining a stripe pattern process, which is an example of an image loss prevention process for preventing the disappearance of an object in an overlapping portion of the imaging space of each of the two cameras.

F13A is a diagram showing an output image portion corresponding to an overlapping portion between the imaging space of the right side camera 2R and the imaging space of the rear camera 2B, and corresponds to the rectangular region R13 shown by the dotted line in FIG.

Further, in F13A, the area PR1 filled with gray is an image area in which the input image portion of the rear camera 2B is arranged, and the input image of the rear camera 2B is set at each coordinate on the output image plane corresponding to the area PR1. Coordinates on the plane are associated.

On the other hand, the area PR2 filled with white is an image area in which the input image portion of the right side camera 2R is arranged, and the input image plane of the right side camera 2R is set at each coordinate on the output image plane corresponding to the area PR2. The above coordinates are associated.

In this embodiment, the region PR1 and the region PR2 are arranged so as to form a striped pattern (striped pattern treatment), and the boundary line of the portion where the region PR1 and the region PR2 are alternately arranged in a striped pattern is the swivel of the excavator 60. It is defined by concentric circles on the horizontal plane centered on the center.

F13B is a top view showing the state of the space area diagonally rearward to the right of the excavator 60, and shows the current state of the space area imaged by both the rear camera 2B and the right side camera 2R. Further, F13B indicates that a rod-shaped three-dimensional object OB exists diagonally to the right and rearward of the excavator 60.

F13C shows a part of the output image generated based on the input image obtained by actually capturing the spatial region indicated by F13B with the rear camera 2B and the right side camera 2R.

Specifically, in the image OB1, the image of the three-dimensional object OB in the input image of the rear camera 2B extends in the extension direction of the line connecting the rear camera 2B and the three-dimensional object OB by the viewpoint conversion for generating the road surface image. Represents what has been done. That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated by using the input image of the rear camera 2B.

Further, in the image OB2, the image of the three-dimensional object OB in the input image of the right-side camera 2R is extended in the extension direction of the line connecting the right-side camera 2R and the three-dimensional object OB by the viewpoint conversion for generating the road surface image. Represents an image. That is, the image OB2 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated by using the input image of the right-side camera 2R.

As described above, in the overlapping portion, the image generation device 100 has the region PR1 to which the coordinates on the input image plane of the rear camera 2B are associated and the region PR2 to which the coordinates on the input image plane of the right camera 2R are associated. To mix. As a result, the image generation device 100 displays both the two images OB1 and the image OB2 relating to one three-dimensional object OB on the output image, and prevents the three-dimensional object OB from disappearing from the output image.

FIG. 14 is a comparison diagram showing the difference between the output image of FIG. 12 and the output image obtained by applying the image disappearance prevention process (stripe pattern process) to the output image of FIG. The output image of FIG. 12 is shown, and the lower figure of FIG. 14 shows the output image after applying the image disappearance prevention processing (striped pattern processing). While the person disappears in the area R12 surrounded by the alternate long and short dash line in the upper figure of FIG. 14, the person is displayed without disappearing in the area R14 surrounded by the alternate long and short dash line in FIG. 14 lower figure.

Note that the image generation device 100 may apply a mesh pattern processing, an averaging process, or the like instead of the stripe pattern processing to prevent the disappearance of the object in the overlapping portion. Specifically, the image generation device 100 outputs the average value of the corresponding pixel values (for example, the luminance value) in the input images of the two cameras by the averaging process to correspond to the overlapping portion. It is used as the pixel value of the image part. Alternatively, the image generation device 100 performs mesh pattern processing to associate the pixel values in the input image of one camera with the pixel values in the input image of the other camera in the output image portion corresponding to the overlapping portion. Are arranged so as to form a mesh pattern with the area to which is associated with. As a result, the image generation device 100 prevents the object in the overlapping portion from disappearing.

Next, with reference to FIGS. 15 to 17, a process in which the image generation means 11 determines an input image to be used for generating an output image from a plurality of input images based on the determination result of the presence / absence determination means 12 (hereinafter, The "first input image determination process") will be described. Note that FIG. 15 is a diagram showing each input image of the three cameras 2 mounted on the excavator 60 and an output image generated by using the input image, and corresponds to FIG. 12. Further, FIG. 16 is a correspondence table showing a correspondence relationship between the determination result of the presence / absence determination means 12 and the input image used for generating the output image. Further, FIG. 17 is a display example of an output image generated based on the input image determined by the first input image determination process.

As shown in FIG. 15, the image generation device 100 projects the input images of the three cameras 2 onto the plane region R1 and the curved surface region R2 of the spatial model MD, and then reprojects them onto the processing target image plane R3. To generate an image to be processed. Further, the image generation device 100 generates an output image by performing an image conversion process (for example, scale conversion, affine transformation, distortion conversion, viewpoint conversion processing, etc.) on the generated image to be processed. As a result, the image generation device 100 generates a virtual viewpoint image for peripheral monitoring that simultaneously displays an image of the vicinity of the excavator 60 looking down from the sky and an image of the surroundings viewed from the excavator 60 in the horizontal direction.

Further, in FIG. 15, each input image of the left side camera 2L, the rear side camera 2B, and the right side camera 2R shows a state in which three workers are present. The output image also shows a state in which nine workers are present around the excavator 60.

Here, with reference to the correspondence table of FIG. 16, the correspondence relationship between the determination result of the presence / absence determination means 12 and the input image used for generating the output image will be described. The ◯ mark indicates that the presence / absence determination means 12 has determined that a person exists, and the × mark indicates that no person has been determined.

In pattern 1, when it is determined that a person exists only in the left side monitoring space ZL and no person exists in the rear monitoring space ZB and the right side monitoring space ZR, the input image of the left side camera 2L is used. Indicates that an output image is generated. This pattern 1 is adopted, for example, when there are workers (three people in this example) only on the left side of the excavator 60. As shown in the output image D1 of FIG. 17, the image generation means 11 outputs the input image of the left side camera 2L that captures the three workers as an output image as it is. In the following, an output image using the input image as it is will be referred to as a “through image”.

Pattern 2 is output using the input image of the rear camera 2B when it is determined that a person exists only in the rear monitoring space ZB and no person exists in the left side monitoring space ZL and the right side monitoring space ZR. Indicates that an image is generated. This pattern 2 is adopted, for example, when there are workers (three people in this example) only behind the excavator 60. As shown in the output image D2 of FIG. 17, the image generation means 11 outputs the input image of the rear camera 2B that captures the three workers as an output image as it is.

In pattern 3, when it is determined that a person exists only in the right side monitoring space ZR and no person exists in the left side monitoring space ZL and the rear monitoring space ZB, the input image of the right side camera 2R is used. Indicates that an output image is generated. This pattern 3 is adopted, for example, when there are workers (three people in this example) only on the right side of the excavator 60. As shown in the output image D3 of FIG. 17, the image generation means 11 outputs the input image of the right-side camera 2R that captures the three workers as an output image as it is.

In pattern 4, when it is determined that a person exists in the left side monitoring space ZL and the rear monitoring space ZB, and no person exists in the right side monitoring space ZR, the output image is used using all three input images. Indicates that is generated. This pattern 4 is adopted, for example, when there are workers (three in this example, a total of six) on the left side and the rear of the excavator 60. As shown in the output image D4 of FIG. 17, the image generation means 11 outputs a peripheral monitoring virtual viewpoint image that captures six workers generated based on the three input images as an output image.

Pattern 5 is an output image using all three input images when it is determined that a person exists in the rear monitoring space ZB and the right side monitoring space ZR and no person exists in the left side monitoring space ZL. Indicates that is generated. This pattern 5 is adopted, for example, when there are workers (three in this example, a total of six) behind and to the right of the excavator 60. As shown in the output image D5 of FIG. 17, the image generation means 11 outputs a peripheral monitoring virtual viewpoint image that captures six workers generated based on the three input images as an output image.

Pattern 6 is an output image using all three input images when it is determined that a person exists in the left side monitoring space ZL and the right side monitoring space ZR and no person exists in the rear monitoring space ZB. Indicates that is generated. This pattern 6 is adopted, for example, when there are workers (three in this example, a total of six) on the left and right sides of the excavator 60. As shown in the output image D6 of FIG. 17, the image generation means 11 outputs a peripheral monitoring virtual viewpoint image that captures six workers generated based on the three input images as an output image.

In pattern 7, when it is determined that a person exists in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR, an output image is generated using all three input images. Represents. This pattern 7 is adopted, for example, when there are workers (three in this example, a total of nine) on the left side, the rear side, and the right side of the excavator 60. As shown in the output image of FIG. 15, the image generation means 11 outputs a peripheral monitoring virtual viewpoint image that captures nine workers generated based on the three input images as an output image.

In pattern 8, when it is determined that no person exists in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR, an output image is generated using all three input images. Represents that. This pattern 8 is adopted, for example, when there is no worker on the left side, the rear side, and the right side of the excavator 60. As shown in the output image D7 of FIG. 17, the image generation means 11 outputs a peripheral monitoring virtual viewpoint image as an output image, which is generated based on the three input images and projects a state in which no worker exists in the surroundings.

As described above, when it is determined that a person exists in only one of the three monitoring spaces, the image generation means 11 outputs a through image of the corresponding camera as an output image. This is to display the person existing in the monitoring space as large as possible on the display unit 5. On the other hand, when it is determined that a person exists in two or more of the three monitoring spaces, the image generation means 11 outputs a virtual viewpoint image for peripheral monitoring without outputting a through image. This is because it is not possible to display all the people existing around the excavator 60 on the display unit 5 with only one through image, and if the virtual viewpoint image for peripheral monitoring is output, it exists around the excavator 60. This is because all people can be displayed on the display unit 5. Further, when it is determined that no person exists in any of the three monitoring spaces, the image generation means 11 outputs a peripheral monitoring virtual viewpoint image without outputting a through image. This is because there is no person to be enlarged and the object other than the person existing around the excavator 60 can be widely monitored.

Further, when displaying the through image, the image generation means 11 may display a text message that indicates which input image was used.

Next, with reference to FIGS. 18 to 20, another example of the process of determining the input image to be used for generating the output image from the plurality of input images based on the determination result of the presence / absence determination means 12 (hereinafter, "No. 1"). 2 Input image determination process ”.) Will be described. Note that FIG. 18 is a top view of the excavator 60 showing another arrangement example of the person detection sensor 6, and corresponds to FIG. 11. Further, FIG. 19 is a correspondence table showing a correspondence relationship between the determination result of the presence / absence determination means 12 and the input image used for generating the output image, and corresponds to FIG. Further, FIG. 20 is a display example of an output image generated based on the input image determined by the second input image determination process.

In the embodiment shown in FIG. 18, the excavator 60 includes two cameras 2 (right side camera 2R and rear camera 2B) and three person detection sensors 6 (right side person detection sensor 6R, right rear person detection sensor 6BR). It is equipped with a rear person detection sensor 6B). The regions CR and CB shown by the alternate long and short dash lines in FIG. 18 indicate the imaging spaces of the right side camera 2R and the rear camera 2B, respectively. Further, the regions ZR, ZBR, and ZB shown by the dotted lines in FIG. 18 indicate the monitoring spaces of the right side person detection sensor 6R, the right rear person detection sensor 6BR, and the rear person detection sensor 6B, respectively. Further, the region X shown by the shaded hatching in FIG. 18 indicates an overlapping portion of the imaging space CR and the imaging space CB (hereinafter, referred to as “overlapping imaging space X”).

The arrangement example of FIG. 18 is an arrangement example of FIG. 11 in that the monitoring space ZR and the monitoring space ZB do not have overlapping portions and the right rear person detection sensor 6BR having the monitoring space ZBR including the overlapping imaging space X is provided. Is different from.

With this arrangement of the human detection sensor 6, the image generation device 100 can determine whether or not a person exists in the overlapping imaging space X. Then, the image generation device 100 can use the determination result in determining the input image to be used for generating the output image, and can switch the content of the output image more appropriately.

Here, with reference to the correspondence table of FIG. 19, the correspondence relationship between the determination result of the presence / absence determination means 12 and the input image used for generating the output image will be described.

Pattern A is output using the input images of the rear camera 2B and the right camera 2R when it is determined that no person exists in all of the rear monitoring space ZB, the right rear monitoring space ZBR, and the right rear monitoring space ZR. Indicates that an image is generated. This pattern A is adopted, for example, when there is no worker around the excavator 60. As shown in the output image D7 of FIG. 17, the image generation means 11 generates and outputs a peripheral monitoring virtual viewpoint image that shows a state in which no worker exists in the surroundings based on the two input images.

Pattern B is output using the input image of the rear camera 2B when it is determined that a person exists only in the rear monitoring space ZB and no person exists in the right rear monitoring space ZBR and the right side monitoring space ZR. Indicates that an image is generated. This pattern B is adopted, for example, when one worker P1 is present behind the excavator 60. As shown in the output image D10 of FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B that captures the worker P1 as an output image as it is.

In pattern C, when it is determined that a person exists only in the right side monitoring space ZR and no person exists in the rear monitoring space ZB and the right rear monitoring space ZBR, the input image of the right side camera 2R is used. Indicates that an output image is generated. This pattern C is adopted, for example, when one worker P2 is present on the right side of the excavator 60. As shown in the output image D11 of FIG. 20, the image generation means 11 outputs the input image of the right-side camera 2R that captures the worker P2 as an output image as it is.

In pattern D, when it is determined that a person exists only in the right rear monitoring space ZBR and no person exists in the rear monitoring space ZB and the right side monitoring space ZR, the input of the rear camera 2B and the right side camera 2R is performed. Indicates that an output image is generated using the image. This pattern D is adopted, for example, when one worker P3 is present in the overlapping imaging space X on the right rear side of the excavator 60. As shown in the output image D12 of FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B that captures the worker P3 as it is as the first output image (left side in the figure), and captures the same worker P3. The input image of the right-side camera 2R is output as it is as the second output image (right side in the figure).

In pattern E, when it is determined that a person exists in the rear monitoring space ZB and the right rear monitoring space ZBR and no person exists in the right rear monitoring space ZR, the input images of the rear camera 2B and the right side camera 2R are determined. Indicates that an output image is generated using. This pattern E is adopted, for example, when one worker P4 is present behind the excavator 60 and one worker P5 is present in the overlapping imaging space X on the right rear side of the excavator 60. .. As shown in the output image D13 of FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B that captures the worker P4 and the worker P5 as it is as the first output image (left side in the figure), and the worker The input image of the right-side camera 2R that captures only P4 is output as it is as the second output image (right side in the figure).

Pattern F is an input image of the rear camera 2B and the right side camera 2R when it is determined that a person exists in the rear monitoring space ZB and the right side monitoring space ZBR and no person exists in the right rear monitoring space ZBR. Indicates that an output image is generated using. This pattern F is adopted, for example, when one worker P6 is present behind the excavator 60 and another worker P7 is present on the right side of the excavator 60. As shown in the output image D14 of FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B that captures the worker P6 as it is as the first output image (left side in the figure), and captures the worker P7. The input image of the right-side camera 2R is output as it is as the second output image (right side in the figure).

In the pattern G, when it is determined that a person exists in the right rear monitoring space ZBR and the right side monitoring space ZR, and no person exists in the rear monitoring space ZB, the input images of the rear camera 2B and the right side camera 2R are determined. Indicates that an output image is generated using. This pattern G is, for example, when one worker P8 exists in the overlapping imaging space X on the right rear side of the excavator 60 and another worker P9 exists on the right side of the excavator 60. Will be adopted. As shown in the output image D15 of FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B that captures the worker P8 as it is as the first output image (left side in the figure), and the worker P8 and the worker The input image of the right-side camera 2R that captures P9 is output as it is as the second output image (right side in the figure).

The pattern H is an output image using the input images of the rear camera 2B and the right camera 2R when it is determined that a person exists in all of the rear monitoring space ZB, the right rear monitoring space ZBR, and the right rear monitoring space ZR. Indicates that is generated. In this pattern H, for example, one worker P10 exists in the overlapping imaging space X on the right rear side of the excavator 60, another worker P11 exists behind the excavator 60, and the excavator 60 exists. It is adopted when there is yet another worker P12 on the right side of. As shown in the output image D16 of FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B that captures the worker P10 and the worker P11 as it is as the first output image (left side in the figure), and the worker The input image of the right-side camera 2R that captures P10 and the worker P12 is output as it is as the second output image (right side in the figure).

In the second input image determination process, when the image generation means 11 shows a person only in the input image of one camera, the image generation means 11 outputs the through image of the corresponding camera as an output image. This is to display the person existing in the monitoring space as large as possible on the display unit 5. On the other hand, when the image generation means 11 shows a person in any of the input images of the two cameras, the through images of the two cameras are output simultaneously and individually. This is because it is not possible to display all the people around the excavator 60 on the display unit 5 with only one through image, and when the through image is output if the virtual viewpoint image for peripheral monitoring is output. This is because it makes it difficult for the operator to see. Further, when it is determined that no person exists in any of the three monitoring spaces, the image generation means 11 outputs a peripheral monitoring virtual viewpoint image without outputting a through image. This is because there is no person to be enlarged and the object other than the person existing around the excavator 60 can be widely monitored.

Further, when displaying the through image, the image generation means 11 may display a text message that indicates which input image was used, as in the case of the first input image determination process.

Next, with reference to FIG. 21, another display example of the output image generated based on the input image determined by the first input image determination process or the second input image determination process will be described.

As shown in FIG. 21, the image generation means 11 may simultaneously display the peripheral monitoring virtual viewpoint image and the through image. For example, when the worker is behind the excavator 60, the image generation means 11 uses the peripheral monitoring virtual viewpoint image as the first output image and the through image of the rear camera 2B as shown in the output image D20 of FIG. May be displayed as the second output image. In this case, the through image is displayed below the peripheral monitoring virtual viewpoint image. This is to allow the operator of the excavator 60 to intuitively understand that there is an operator behind the excavator 60. Further, when the worker is present on the left side of the excavator 60, the image generation means 11 uses the peripheral monitoring virtual viewpoint image as the first output image as shown in the output image D21 of FIG. 21, and the left side camera 2L. The through image may be displayed as the second output image. In this case, the through image is displayed on the left side of the peripheral monitoring virtual viewpoint image. Similarly, when an operator is present on the right side of the excavator 60, the image generation means 11 uses the peripheral monitoring virtual viewpoint image as the first output image and the right side camera 2R as shown in the output image D22 of FIG. The through image of may be displayed as the second output image. In this case, the through image is displayed on the right side of the peripheral monitoring virtual viewpoint image.

Alternatively, the image generation means 11 may simultaneously display the peripheral monitoring virtual viewpoint image and the plurality of through images. For example, when there are workers behind and to the right of the excavator 60, the image generation means 11 uses the peripheral monitoring virtual viewpoint image as the first output image and the rear camera 2B as shown in the output image D23 of FIG. The through image of the right side camera 2R may be displayed as the second output image, and the through image of the right side camera 2R may be displayed as the third output image. In this case, the through image of the rear camera 2B is displayed below the virtual viewpoint image for peripheral monitoring, and the through image of the right side camera 2R is displayed on the right side of the virtual viewpoint image for peripheral monitoring.

When displaying a plurality of output images at the same time, the image generation means 11 may display information representing the contents of the output images. For example, the image generation means 11 may display text such as a “rear camera through image” on or around the through image of the rear camera 2B.

Further, when displaying a plurality of output images at the same time, the image generation means 11 may pop up one or a plurality of through images on the peripheral monitoring virtual viewpoint image.

With the above configuration, the image generation device 100 switches the content of the output image based on the determination result of the presence / absence determination means 12. Specifically, the image generation device 100, for example, reduces and changes the contents of individual input images and displays a virtual viewpoint image for peripheral monitoring, and a through image that displays the contents of individual input images as they are. Switch. In this way, the image generation device 100 displays a through image when an operator is detected in the vicinity of the excavator 60, so that the operator of the excavator 60 can display only the virtual viewpoint image for peripheral monitoring as compared with the case where the operator displays only the peripheral monitoring virtual viewpoint image. It is possible to more reliably prevent the worker from being overlooked. This is because the operator is displayed in a large and easy-to-understand manner.

Further, in the above-described embodiment, when the image generation means 11 generates an output image based on the input image of one camera, the through image of that camera is used as the output image. However, the present invention is not limited to this configuration. For example, the image generation means 11 may generate a rear monitoring virtual viewpoint image as an output image based on the input image of the rear camera 2B.

Further, in the above-described embodiment, the image generation device 100 associates the monitoring space of one person detection sensor with the imaging space of one camera, but provides the monitoring space of one person detection sensor with the imaging space of a plurality of cameras. It may be made to correspond, and the monitoring space of a plurality of person detection sensors may correspond to the imaging space of one camera.

Further, in the above-described embodiment, the image generation device 100 overlaps a part of the monitoring spaces of the two adjacent human detection sensors, but the monitoring spaces may not be overlapped. Further, the image generation device 100 may completely include the monitoring space of one person detection sensor in the monitoring space of another person detection sensor.

Further, in the above-described embodiment, the image generation device 100 switches the content of the output image at the moment when the determination result of the presence / absence determination means 12 changes. However, the present invention is not limited to this configuration. For example, the image generation device 100 may set a predetermined delay time from the change of the determination result of the presence / absence determination means 12 to the switching of the content of the output image. This is to prevent the contents of the output image from being frequently switched.

Next, with reference to FIGS. 22 and 23, the alarm control means 13 controls the alarm output unit 7 based on the determination result of the presence / absence determination means 12 (hereinafter, referred to as “alarm control process”). explain. Note that FIG. 22 is a flowchart showing the flow of the alarm control process, and FIG. 23 is an example of the transition of the output image displayed during the alarm control process. Further, the alarm control means 13 repeatedly executes this alarm control process at a predetermined cycle. Further, the image generation device 100 is mounted on the excavator 60 shown in FIG.

First, the presence / absence determination means 12 determines whether or not there is a person around the excavator 60 (step S11). At this time, the image generation means 11 generates and displays, for example, a peripheral monitoring virtual viewpoint image as shown in the output image D31 of FIG. 23.

When it is determined that a person exists around the excavator 60 (YES in step S11), the presence / absence determination means 12 is any of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR. It is determined whether or not there is a person in (step S12).

Then, when it is determined that a person exists only in the left side monitoring space ZL (the left side in step S12), the person presence / absence determination means 12 outputs the left side detection signal to the alarm control means 13. Then, the alarm control means 13 that has received the left side detection signal outputs an alarm start signal to the left side alarm output unit 7L, and outputs an alarm from the left side alarm output unit 7L (step S13). Further, the image generation means 11 displays, for example, a through image of the left side camera 2L as shown in the output image D32 of FIG. 23 as an output image. Alternatively, as shown in the output image D33 of FIG. 23, for example, the image generation means 11 uses the peripheral monitoring virtual viewpoint image as the first output image (the image on the right side in the figure) and the through image of the left camera 2L as the first output image. 2 It may be displayed as an output image (the image on the left side in the figure).

Further, when it is determined that a person exists only in the rear monitoring space ZB (behind step S12), the person presence / absence determining means 12 outputs a rearward detection signal to the alarm control means 13. Then, the alarm control means 13 that has received the rearward detection signal outputs an alarm start signal to the rearward alarm output unit 7B, and outputs an alarm from the rearward alarm output unit 7B (step S14). Further, the image generation means 11 displays, for example, a through image of the rear camera 2B as an output image. Alternatively, the image generation means 11 may display the peripheral monitoring virtual viewpoint image as the first output image and the through image of the rear camera 2B as the second output image.

Further, when it is determined that a person exists only in the right side monitoring space ZR (the right side in step S12), the person presence / absence determining means 12 outputs a right side detection signal to the alarm control means 13. Then, the alarm control means 13 that has received the right side detection signal outputs an alarm start signal to the right side alarm output unit 7R, and outputs an alarm from the right side alarm output unit 7R (step S15). Further, the image generation means 11 displays, for example, a through image of the right-side camera 2R as an output image. Alternatively, the image generation means 11 may display the peripheral monitoring virtual viewpoint image as the first output image and the through image of the right side camera 2R as the second output image.

On the other hand, when it is determined that no person exists around the excavator 60 (NO in step S11), the presence / absence determination means 12 performs the alarm control process this time without outputting a detection signal to the alarm control means 13. finish.

When the alarm control means 13 determines that a person exists in two or more of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR by the person presence / absence determining means 12. Outputs an alarm from two or more corresponding alarm output units of the left side alarm output unit 7L, the rear alarm output unit 7B, and the right side alarm output unit 7R. Further, when the image generation means 11 determines that a person exists in two or more of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR by the person existence determination means 12. Displays the output image in a manner as described in the above embodiment. Specifically, the image generation means 11 may display only the peripheral monitoring virtual viewpoint image as shown in the output image D34 of FIG. 23. Further, the image generation means 11 may simultaneously display two or more through images as shown in the output image D35 of FIG. Further, the image generation means 11 may simultaneously display the peripheral monitoring virtual viewpoint image and two or more through images as shown in the output image D36 of FIG. 23.

Further, in the configuration in which the image generation device 100 outputs an alarm sound from the alarm output unit 7, the contents (high and low) of the alarm sounds of the left side alarm output unit 7L, the rear alarm output unit 7B, and the right side alarm output unit 7R are obtained. , Output interval, etc.) may be different. Similarly, in the configuration in which the image generation device 100 outputs light from the alarm output unit 7, the light contents (color, color, respectively) of the left side alarm output unit 7L, the rear alarm output unit 7B, and the right side alarm output unit 7R are used. The light emission interval, etc.) may be different. This is so that the operator of the excavator 60 can more intuitively recognize the rough position of a person existing around the excavator 60 due to the difference in the content of the alarm.

With the above configuration, the image generation device 100 enables the operator of the excavator 60 to intuitively grasp the rough position of the operator existing around the excavator 60. For example, even if the image generator 100 does not detect the exact position of the worker, it only needs to determine whether the worker is on the left side, the rear side, or the right side of the excavator 60, and the determined direction. Can be intuitively conveyed to the operator of the excavator 60.

In the above-described embodiment, the alarm output unit 7 is composed of three independent buzzers, but the sound may be localized by using a surround system including a plurality of speakers.

Although the preferred examples of the present invention have been described in detail above, the present invention is not limited to the above-mentioned examples, and various modifications and substitutions are made to the above-mentioned examples without departing from the scope of the present invention. Can be added.

For example, in the above embodiment, the image generation device 100 adopts a cylindrical space model MD as a space model, but a space model having another columnar shape such as a polygonal prism may be adopted, and the bottom surface and the space model may be adopted. A space model composed of two side surfaces may be adopted, or a space model having only side surfaces may be adopted.

Further, the image generation device 100 is mounted on a self-propelled excavator equipped with movable members such as a bucket, an arm, a boom, and a swivel mechanism together with a camera and a human detection sensor. Then, the image generation device 100 constitutes an operation support system that supports the movement of the excavator and the operation of the movable members while presenting the surrounding image to the operator. However, the image generator 100 may be mounted together with a camera and a person detection sensor on a work machine that does not have a swivel mechanism such as a forklift or an asphalt finisher. Alternatively, the image generator 100 may be mounted together with a camera and a human detection sensor on a work machine having a movable member but not self-propelled, such as an industrial machine or a fixed crane. Then, the image generation device 100 may configure an operation support system that supports the operation of these work machines.

Further, although the peripheral monitoring device has been described by taking the image generation device 100 including the camera 2 and the display unit 5 as an example, the peripheral monitoring device may be configured as a device that does not include the image display function by the camera 2, the display unit 5, and the like. For example, as shown in FIG. 24, the peripheral monitoring device 100A as a device that executes the alarm control process includes a camera 2, an input unit 3, a storage unit 4, a display unit 5, a coordinate matching means 10, and an image generating means 11. May be omitted.

1 ... Control unit 2 ... Camera 2L ... Left side camera 2R ... Right side camera 2B ... Rear camera 3 ... Input unit 4 ... Storage unit 5 ... Display unit 6 ...・ Person detection sensor 6L ・ ・ ・ Left side person detection sensor 6R ・ ・ ・ Right side person detection sensor 6B ・ ・ ・ Rear person detection sensor 7 ・ ・ ・ Alarm output unit 7L ・ ・ ・ Left side alarm output unit 7B ・ ・ ・Rear alarm output unit 7R ・ ・ ・ Right side alarm output unit 10 ・ ・ ・ Coordinate mapping means 11 ・ ・ ・ Image generation means 12 ・ ・ ・ Presence / absence determination means 13 ・ ・ ・ Alarm control means 40 ・ ・ ・ Input image ・Spatial model compatible map 41 ... Spatial model / processed target image compatible map 42 ... Processed target image / output image compatible map 60 ... Excavator 61 ... Lower traveling body 62 ... Turning mechanism 63 ... Upper swivel body 64 ・ ・ ・ Cab 100 ・ ・ ・ Image generator 100A ・ ・ ・ Peripheral monitoring device

In order to achieve the above object, the excavator according to the embodiment of the present invention is attached to the lower traveling body, the upper rotating body rotatably mounted on the lower traveling body, and the upper rotating body, and is attached to the attachment. The left and right parts of the upper swivel body so as to image the included boom, the arm attached to the boom and included in the attachment, and the left, right, and rear three directions of the upper swivel body. , And three image pickup devices mounted at three locations at the rear, an cab mounted on the upper swing body, and a display unit installed in the cab, and the display unit is provided with the three directions. at least a one of the first image corresponding to the direction, is generated by combining the image of the three-way, regardless of the orientation of the lower traveling body, left before Symbol upper rotating body, right out of the , And the rear area are displayed at the same time, the upper swivel body rotates about a point corresponding to the swivel axis when the upper swivel body turns with respect to the lower traveling body, and the second image of the upper swivel body looking down from the sky. And are displayed at the same time.

Claims (6)

With the lower running body,
An upper swing body that is freely mounted on the lower running body and
The boom attached to the upper swing body and included in the attachment,
An arm attached to the boom and included in the attachment,
Three imaging devices mounted on the left, right, and rear parts of the upper swivel body so as to image the left, right, and rear three directions of the upper swivel body.
The driver's cab mounted on the upper swing body and
A display unit installed in the driver's cab is provided.
On the display
The first image corresponding to at least one of the three directions and
The upper swivel body, which is generated by synthesizing the images in the three directions and simultaneously displays at least the left, right, and rear regions of the upper swivel body regardless of the orientation of the lower traveling body. The second image looking down from the sky and
At the same time,
Excavator. The presence or absence of an object entering is determined by at least one of the image pickup device or a sensor provided separately from the image pickup device, and the first image corresponding to the direction in which the object has entered is placed next to the second image. To display
The excavator according to claim 1. The image used for the first image is switched based on the determination result of the presence or absence of the invasion of the object.
The excavator according to claim 2. When the first image is an image corresponding to the right side of the upper swivel body, the first image is displayed on the display unit so as to be located on the right side of the second image.
The excavator according to any one of claims 1 to 3. When the first image is an image corresponding to the left side of the upper swivel body, the first image is displayed on the display unit so as to be located on the left side of the second image.
The excavator according to any one of claims 1 to 3. When the first image is an image corresponding to the rear of the upper swivel body, the first image is displayed on the display unit so as to be located below the second image.
The excavator according to any one of claims 1 to 3.