JP2023106227A - Depth information processing device, depth distribution estimation method, depth distribution detection system, and trained model generation method - Google Patents

Abstract

To reduce the distortion and displacement of a depth distribution attributable to a variation in the measurement time of a distance measuring device.SOLUTION: A depth information processing device in an embodiment of the present disclosure comprises a distance information acquisition unit, an image acquisition unit, and a processor. The distance information acquisition unit acquires output data from a scanning type distance measuring device. The image acquisition unit acquires an image in which a range overlapping the detection range of the distance measuring device is imaged. The processor estimates, on the basis of the output data and image, a depth distribution at a prescribed timing in a first period which is the one frame period of the distance measuring device. In order to estimate the depth distribution, the processor performs a plurality of processing to generate composite depth information from the output data, for each of a plurality of second periods into which the first period is divided in time. The processor inputs the generated composite depth information to a trained model so as to estimate a depth distribution.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to a depth information processing device, a depth distribution estimation method, a depth distribution detection system, and a trained model generation method.

A scanning range finder, such as a laser radar, is used to measure the azimuth and distance to an object. For example, a technology has been disclosed that uses a laser radar to measure the distance and direction to objects around the vehicle and determines the placement of parked vehicles based on the obtained data point sequence (see, for example, Patent Document 1). ).

JP-A-2002-243857

However, since a normal laser radar performs sequential laser scanning, the resolution and frame rate are lower than those of camera images. In addition, in a scanning laser radar, if the number of measurement points is increased in order to increase the resolution, the scanning time will become longer. Furthermore, when the object to be detected is a moving object, the moving object moves during scanning. For this reason, due to the difference in measurement time during the measurement of one frame, the depth distribution may be distorted and deviated, making accurate measurement impossible.

Therefore, the object of the present disclosure, which focuses on these points, is to reduce the distortion and deviation of the depth distribution caused by the difference in measurement time in measuring the depth distribution using a scanning rangefinder. be.

A depth information processing apparatus according to an embodiment of the present disclosure includes a distance information acquisition section, an image acquisition section, and a processor. The distance information acquisition unit acquires output data from a scanning rangefinder. The image acquisition unit acquires an image of a range overlapping the detection range of the distance measuring device. Based on the output data and the image, the processor estimates a depth distribution at predetermined timing within a first period, which is a period of one frame of the range finder. The processor corresponds, from the output data, the depth, the two-dimensional position where the depth is detected, and the second period for each of a plurality of second periods obtained by temporally dividing the first period. Generate multiple pieces of depth information including time information for The processor generates area division information for dividing the detection range into a background area and a moving object area for each of the second periods from the image. For each of the second periods, the processor includes accumulated depth information obtained by accumulating the plurality of depth information in the first period in the background region, and the plurality of depth information in the second period. Composite depth information is generated that includes depth information in the region of the moving object. The processor further receives a plurality of synthetic depth information of the second period as input and learns a trained model using teacher data outputting the depth distribution at the predetermined timing within the first period. , the depth distribution is estimated by inputting the generated synthetic depth information.

A depth distribution estimation method according to an embodiment of the present disclosure is a method executed by a processor of a computer. The method includes acquiring output data from a scanning rangefinder, and acquiring an image of a range overlapping a detection range of the rangefinder. According to the method, from the output data, for each of the second periods, depth and generating a plurality of pieces of depth information including time information corresponding to the position in the two-dimensional direction where the depth was detected and the second period. The method includes generating, from the image, segmentation information segmenting a background region and a moving object region of the detection range for each of the second time periods. The method includes, for each of the second periods, including accumulated depth information obtained by accumulating the plurality of depth information in the first period in the region of the background; generating synthetic depth information including depth information in the region of the moving object. In the method, a trained model trained using teacher data having as input a plurality of synthetic depth information of the second period and outputting a depth distribution at a predetermined timing within the first period is generated. estimating the depth distribution by inputting the synthesized depth information obtained by the method.

A depth distribution detection system according to an embodiment of the present disclosure includes a scanning rangefinder, an imaging device,
and a depth information processor. The imaging device captures an image of a range that overlaps with the detection range of the distance measuring device. The depth information processing device calculates a depth distribution at a predetermined timing within a first period, which is a period of one frame of the range finder, based on the output data of the range finder and the image captured by the imaging device. Contains a computing processor. The processor corresponds, from the output data, the depth, the two-dimensional position where the depth is detected, and the second period for each of a plurality of second periods obtained by temporally dividing the first period. Generate multiple pieces of depth information including time information for The processor generates area division information for dividing the detection range into a background area and a moving object area for each of the second periods from the image. For each of the second periods, the processor includes accumulated depth information obtained by accumulating the plurality of depth information in the first period in the background region, and the plurality of depth information in the second period. Composite depth information is generated that includes depth information in the region of the moving object. The processor receives a plurality of synthetic depth information of the second period as input, and trains a trained model using teacher data outputting the depth distribution at the predetermined timing within the first period, The depth distribution is estimated by inputting the generated synthetic depth information.

A learned model generation method according to an embodiment of the present disclosure is a computer-executed learned model generation method. The method includes a plurality of depth information and area division information generated for each of a plurality of second periods obtained by temporally dividing a first period, which is a period of one frame of a scanning ranging device, and A plurality of pieces of data combining depth distributions at predetermined timings within the first period are acquired or generated. Each of the depth information includes depth, a position in the two-dimensional direction where the depth is detected, and time information corresponding to the second period, and the area division information is a background area of the detection range of the rangefinder. and moving object regions. The method includes, for each of the second periods, including accumulated depth information obtained by accumulating the plurality of depth information in the first period in the region of the background; Composite depth information is generated that includes depth information in the region of the moving object. The method receives the synthetic depth information of the plurality of second periods as input and generates a trained model by performing machine learning using teacher data having the depth distribution as output.

According to the embodiments of the present disclosure, it is possible to reduce the distortion and deviation of the depth distribution due to the deviation of the measurement time in measuring the depth distribution using the scanning rangefinder.

1 is a configuration diagram showing a basic configuration of a depth distribution detection system according to an embodiment of the present disclosure; FIG. 2 is a configuration diagram showing a more detailed example of the depth distribution detection system of FIG. 1; FIG. 2 is a block diagram showing a schematic configuration of the depth information processing device of FIG. 1; FIG. 2 is a flowchart showing processing executed by the depth information processing apparatus of FIG. 1; 2 is a diagram showing an example of a scene detected by the distance measuring device and imaging device of FIG. 1; FIG. 2 is a diagram illustrating an image and depth information acquired by a depth information processing device during one-frame ranging period of the ranging device of FIG. 1; FIG. It is a figure explaining the method to generate|occur|produce area division information. FIG. 10 is a diagram illustrating a method of generating synthetic depth information; 1 is a configuration diagram showing a schematic configuration of a learning computer that generates a trained model; FIG. 4 is a flow chart showing processing for generating a trained model by a learning computer;

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The figures used in the following description are schematic. The dimensions, ratios, etc. on the drawings do not necessarily match the actual ones.

A depth distribution detection system 10 according to an embodiment of the present disclosure includes a distance measuring device 11, an imaging device 12, and a depth information processing device 13, as shown in FIG. In addition, in this application, the depth means the distance from the distance measuring device 11 to the object. In this application, depth is used synonymously with distance.

The rangefinder 11 is a scanning type rangefinder such as a laser radar. A laser radar irradiates a detection range with laser light and measures the time it takes for the reflected laser light to return. Laser radar is also called LiDAR (Light Detection And Ranging). The imaging device 12 has an imaging optical system and an imaging device, and acquires an image of the detection range. The imaging device 12 is configured to capture an image of a range that overlaps with the detection range of the distance measuring device 11 .

The distance measuring device 11 and the imaging device 12 can be configured so that the coordinate systems for observing the object are substantially the same or close to each other. Therefore, the distance measuring device 11 and the imaging device 12 may be configured such that the optical axis ax of the optical system 14 for detecting or imaging the subject is aligned. The optical system 14 includes various configurations. Further, the distance measuring device 11 and the imaging device 12 may be arranged so that the optical axes of the optical systems for detecting or imaging a subject are parallel and close to each other. In this case, the optical system 14 for matching the optical axes of the distance measuring device 11 and the imaging device 12 may be omitted. Between the distance measuring device 11 and the imaging device 12, there is no or little parallax between detected data and images. The depth output from the distance measuring device 11 and the image output from the imaging device 12 can be superimposed so that their positions correspond.

The distance measuring device 11 and the imaging device 12 are arranged fixedly with respect to the background. Therefore, the distance measuring device 11 and the imaging device 12 may be fixed with respect to a stationary object. For example, when the depth distribution detection system 10 is used outdoors, the distance measuring device 11 and the imaging device 12 may be fixed to a structure or the like provided on the roadside or above the road.

The depth information processing device 13 is a computer. The depth information processing device 13 may be any of various computers including general-purpose computers, workstations, and PCs (Personal Computers). The depth information processing device 13 may read a dedicated program and data and execute each function of the depth information processing device 13 described below.

The output data from the distance measuring device 11 acquired by the depth information processing device 13 includes at least two-dimensional position information indicating the direction of the object to be measured and distance information. The two-dimensional positional information corresponds to the position on the image captured by the imaging device 12 . The output data may further include information on the time when the depth was measured. The depth information processing device 13 further acquires an image captured by the imaging device 12 from the imaging device 12 .

Based on the output data output from the distance measuring device 11 and the image output from the imaging device 12, the depth information processing device 13 determines a predetermined depth within a first period _T1 , which is a period of one frame of the distance measuring device 11. Estimate the depth distribution at the timing of In the present application, "depth distribution" represents a two-dimensional depth distribution when the detection range observed by the distance measuring device 11 is viewed in plan. "Depth distribution" is used synonymously with "depth map". The "depth distribution" need not be represented as an image. The predetermined timing can be, for example, the center time of one frame of the distance measuring device 11 . The depth distribution estimated by the depth information processing device 13 has reduced distortion and deviation due to the difference in measurement time of the distance measuring device 11 . The depth information processing device 13 has a learned model by machine learning, and uses the output data output from the distance measuring device 11 and the image output from the imaging device 12 to estimate the depth distribution by machine learning. can be done.

A more specific example of the depth distribution detection system 10 will be described with reference to FIG. The depth distribution detection system 10 includes a detection device 21 that includes the functions of the distance measurement device 11 and the imaging device 12 and a depth information processing device 13 . The detection device 21 includes an irradiation unit 22 , a reflection unit 23 , a control unit 24 , a first optical system 25 and a detection element 26 as components corresponding to the distance measurement device 11 . The detection device 21 also includes a control unit 24 , a second optical system 27 , and an imaging device 28 as components corresponding to the imaging device 12 . The detection device 21 further includes a switching section 32 .

(Obtain depth information)
The irradiation unit 22 radiates at least one electromagnetic wave of infrared rays, visible rays, ultraviolet rays, and radio waves. In one embodiment, the irradiation section 22 emits infrared rays. The irradiating unit 22 irradiates the radiated electromagnetic wave toward the object ob directly or indirectly via the reflecting unit 23 . In the embodiment illustrated in FIG. 2 , the irradiation unit 22 indirectly irradiates the radiated electromagnetic wave toward the object ob through the reflection unit 23 .

In one embodiment, the irradiator 22 emits a beam of electromagnetic waves with a narrow width, for example, 0.5°. In one embodiment, the irradiation unit 22 can emit electromagnetic waves in pulses. For example, the irradiation unit 22 includes LEDs (Light Emitting Diodes) and LDs (Laser Diodes). The irradiation unit 22 switches between electromagnetic wave emission and stop based on the control of the control unit 24, which will be described later.

The reflecting unit 23 changes the irradiation position of the electromagnetic wave with which the object ob is irradiated by reflecting the electromagnetic wave emitted from the irradiation unit 22 while changing the direction of the electromagnetic wave. The irradiation position is equal to the detection position where the distance measuring device 11 measures the distance. The reflector 23 scans the object ob with the electromagnetic waves emitted from the irradiator 22 . Therefore, the detection element 26 cooperates with the reflection section 23 to form a scanning range sensor.

The reflector 23 can scan the irradiation position of the electromagnetic wave in two-dimensional directions within the detection range. The reflecting unit 23 can perform regular scanning in which the irradiation position detected by the distance measuring device 11 is shifted in the vertical direction sequentially while scanning in the horizontal direction, like a progressive scanning method. In addition, the reflecting unit 23 may sequentially set the irradiation position of the electromagnetic wave to a random position in the two-dimensional direction. In the present disclosure, “scanning” also includes sequential measurements while changing irradiation positions at random. Reflector 23 may scan the detection range in any other manner.

The reflecting section 23 is configured such that at least part of the irradiation area of the electromagnetic waves emitted and reflected from the irradiating section 22 is included in the detection range of the detecting element 26 . Therefore, at least a part of the electromagnetic waves irradiated to the object ob through the reflector 23 can be detected by the detection element 26 .

The reflector 23 includes, for example, a MEMS (Micro Electro Mechanical Systems) mirror, a polygon mirror, a galvanomirror, and the like.

The reflector 23 changes the direction in which the electromagnetic waves are reflected under the control of the controller 24, which will be described later. Further, the reflection unit 23 may have an angle sensor such as an encoder, and may notify the control unit 24 of the angle detected by the angle sensor as direction information for reflecting the electromagnetic waves. With such a configuration, the controller 24 can calculate the irradiation position based on the direction information acquired from the reflector 23 . In addition, the control unit 24 can calculate the irradiation position based on the drive signal input to the reflecting unit 23 to change the direction in which the electromagnetic waves are reflected.

When the control unit 24 calculates the irradiation position, that is, the detection position for detecting the depth, based on the direction information acquired from the reflection unit 23 or the drive signal input to the reflection unit 23, the reflection unit 23 scans the electromagnetic waves. The center axis in the direction to the optical axis ax can be arranged substantially parallel and close to the optical axis ax. In this case, the detection axis by the distance measuring device 11 is the central axis along which the reflector 23 scans the electromagnetic waves.

Control unit 24 includes one or more processors and memory. The processor may include at least one of a general-purpose processor that loads a specific program and executes a specific function, and a dedicated processor that specializes in specific processing. A dedicated processor may include an Application Specific Integrated Circuit (ASIC). The processor may include a programmable logic device (PLD). A PLD may include an FPGA (Field-Programmable Gate Array). The control unit 24 may include at least one of SoC (System-on-a-Chip) and SiP (System In a Package) in which one or more processors cooperate.

The control unit 24 is configured to be able to control the reflection unit 23 and the switching unit 32 . The control unit 24 can control the switching unit 32 so that the detecting element 26 can acquire the reflected electromagnetic wave according to the irradiation position and the irradiation time of the electromagnetic wave by the reflecting unit 23 . As will be described later, the control unit 24 can acquire detection information from the detection elements 26 and generate depth information. Also, the control unit 24 can acquire an image signal from the imaging device 28 . The control unit 24 can output depth information and image signals to the depth information processing device 13 .

The first optical system 25 is irradiated from the irradiation unit 22 and reflected by the reflection unit 23, so that the detection element 26 detects the reflected wave from the target ob of the electromagnetic waves irradiated toward the detection range. proceed as follows.

The detection element 26 includes an element capable of detecting electromagnetic waves emitted from the irradiation section 22 . For example, the sensing element 26 includes a single element such as an Avalanche PhotoDiode (APD), a PhotoDiode (PD) and a ranging image sensor. Detector elements 26 may include element arrays such as APD arrays, PD arrays, ranging imaging arrays, and ranging image sensors. In one embodiment, the detection element 26 transmits detection information indicating detection of the reflected wave from the subject to the control section 24 as a signal. The detection element 26 detects, for example, electromagnetic waves in the infrared band.

Note that the detection element 26 is a single element that constitutes the distance measuring sensor described above, so long as it can detect electromagnetic waves, and the object ob need not be imaged on the detection surface. Therefore, the detection element 26 does not have to be provided at the secondary imaging position, which is the imaging position by the first post-stage optical system 30 . That is, in this configuration, if the detecting element 26 is at a position where electromagnetic waves from all angles of view can be incident on the detecting surface, the detecting element 26 travels in the first direction d1 by the switching unit 32, and then moves toward the first post-optical system. It may be placed anywhere on the path of the electromagnetic wave traveling through 30 .

The control unit 24 acquires depth based on the electromagnetic waves detected by the detection element 26 . Based on the detection information detected by the detection element 26, the control unit 24 acquires the depth of the irradiation position irradiated by the irradiation unit 22 by a ToF (Time-of-Flight) method as described below.

By inputting an electromagnetic wave radiation signal to the irradiation unit 22, the control unit 24 causes the irradiation unit 22 to radiate pulsed electromagnetic waves. The irradiation unit 22 irradiates an electromagnetic wave based on the input electromagnetic wave radiation signal. The electromagnetic waves radiated by the irradiation unit 22 and reflected by the reflection unit 23 to irradiate an arbitrary irradiation area are reflected in the irradiation area. When the detection element 26 detects the electromagnetic waves reflected in the irradiation area, the detection element 26 notifies the control unit 24 of detection information.

The control unit 24 measures the time from the time when the irradiation unit 22 emits the electromagnetic waves to the time when the detection information is acquired. The control unit 24 multiplies the time by the speed of light and divides by 2 to calculate the distance to the irradiation position. When the detection element 26 is a single element, the control unit 24 calculates the irradiation position based on the direction information acquired from the reflection unit 23 or the drive signal output to the reflection unit 23 by itself, as described above. do. When the detection element 26 includes an array of elements, the control unit 24 can calculate the irradiation position from the position at which the electromagnetic wave reflected by the object ob is detected on the element array. In this case, the irradiation position can be specified on substantially the same coordinate axis as the image obtained from the imaging device 28 with no parallax. The control unit 24 calculates the distance to each irradiation position while changing the irradiation position, thereby creating output data including information on depth and detection position.

(Acquisition of image information)
The second optical system 27 forms an image of the object ob in the detection range of the detection element 26 , that is, the range overlapping the detection range of the distance measuring device 11 on the detection surface of the imaging element 28 .

The imaging device 28 converts the image formed on the detection surface into an electrical signal to generate an image of the detection range including the object ob. The imaging device 28 may include either a CCD image sensor (Charge-Coupled Device Image Sensor) or a CMOS image sensor (Complementary MOS Image Sensor).

The imaging device 28 outputs the generated image to the control section 24 . The control unit 24 may perform arbitrary processing such as distortion correction, brightness adjustment, contrast adjustment, gamma correction, etc. on the image. If there is a deviation between the coordinate system of the output data output from the detection element 26 and the coordinate system of the image output from the imaging element 28, the control unit 24 adjusts them so that they are closer to each other. good.

(Configuration of optical system and switching unit)
The first optical system 25 includes a front optical system 29 that is common to the second optical system 27 and a first rear optical system 30 that is positioned behind the switching unit 32 . The second optical system 27 includes a pre-stage optical system 29 that is common to the first optical system 25 and a second post-stage optical system 31 positioned after the switching section 32 . The pre-stage optical system 29 includes, for example, at least one of a lens and a mirror, and forms an image of an object ob, which is a subject.

The switching unit 32 is provided at or in the vicinity of the primary imaging position, which is the imaging position of the image of the object ob, which is separated from the preceding optical system 29 by the preceding optical system 29, by the preceding optical system 29. It is good if there is The switching unit 32 has an action surface as on which the electromagnetic wave that has passed through the pre-stage optical system 29 is incident. The active surface as is composed of a plurality of pixels px arranged two-dimensionally. The action surface as is a surface that causes an electromagnetic wave to have an action such as reflection or transmission in at least one of a first state and a second state described later.

The switching unit 32 can switch for each pixel px between a first state in which an electromagnetic wave incident on the action surface as travels in a first direction d1 and a second state in which it travels in a second direction d2. be. A first state is a first reflection state in which an electromagnetic wave incident on the working surface as is reflected in a first direction d1. The second state is a second reflection state in which the electromagnetic wave incident on the action surface as is reflected in the second direction d2.

More specifically, the switching unit 32 includes a reflecting surface that reflects electromagnetic waves for each pixel px. The switching unit 32 switches between the first reflection state and the second reflection state for each pixel px by changing the orientation of the reflection surface for each pixel px. In one embodiment, the switching unit 32 includes, for example, a DMD (Digital Micro mirror Device). The DMD drives the minute reflective surfaces that make up the active surface as so that the reflective surface is tilted at a predetermined angle, for example +12° or −12°, with respect to the active surface as for each pixel px. can be switched to The active surface as is parallel to the surface of the substrate on which the minute reflecting surface of the DMD is placed.

The switching unit 32 switches between the first state and the second state for each pixel px under the control of the control unit 24 . For example, the switching unit 32 can simultaneously switch some of the pixels px1 to the first state so that the electromagnetic waves incident on the pixels px1 travel in the first direction d1, and switch some of the pixels px2 to the first state. By switching to state 2, the electromagnetic waves incident on the pixel px2 can be caused to travel in the second direction d2. Further, the switching unit 32 switches the same pixel px from the first state to the second state, thereby directing the electromagnetic waves incident on the pixel px in the first direction d1 and then in the second direction d2. can proceed.

As shown in FIG. 2 , the first post-stage optical system 30 is provided in the first direction d1 from the switching section 32 . The first post-stage optical system 30 includes, for example, at least one of a lens and a mirror. The first post-stage optical system 30 causes the electromagnetic wave whose traveling direction is switched by the switching unit 32 to enter the detection element 26 .

The second post-stage optical system 31 is provided in the second direction d2 from the switching section 32 . The second post-stage optical system 31 includes, for example, at least one of a lens and a mirror. The second post-stage optical system 31 forms an image of the target ob as an electromagnetic wave whose traveling direction is switched by the switching unit 32 on the detection surface of the imaging device 28 .

With the configuration as described above, the detection device 21 aligns the optical axis of the front optical system 29 with the optical axis of the first rear optical system 30 in the first direction d1 in which electromagnetic waves travel in the first state, and In state 2, it is possible to match the optical axis of the second post-stage optical system 31 in the second direction d2 in which the electromagnetic wave travels. Therefore, by switching the pixel px of the switching unit 32 between the first state and the second state, the detection device 21 can detect the depth detected by the detection element 26 in the case of an element array and the depth detected by the image pickup element 28. It is possible to reduce the disparity of parallax with the image to be displayed.

The control unit 24 can switch some of the pixels px in the switching unit 32 to the first state and switch some other pixels px to the second state as the reflecting unit 23 moves the irradiation position. Therefore, the detection device 21 can cause the detection element 26 to detect electromagnetic waves in some pixels px and simultaneously cause the imaging element 28 to detect images in another part of pixels px. As a result, the detection device 21 can substantially simultaneously acquire the depth within the same field of view and the image of the portion excluding the vicinity of the irradiation position of the electromagnetic waves.

In addition, in the configuration example of FIG. 2, an optical system including the switching unit 32 is used in order to make the detection optical axis of the distance measuring device 11 and the optical axis of the imaging device 12 match or approach each other. However, the method for making the optical axes of the distance measuring device 11 and the imaging device 12 coincide or approach each other is not limited to this. For example, the optical system 14 in FIG. 1 utilizes the difference in wavelength between the light detected by the distance measuring device 11 and the image captured by the imaging device 12, and uses a dichroic mirror or a dichroic prism to convert the optical power of the distance measuring device 11. The optical axis of the system and the optical axis of the imaging device 12 can be substantially aligned.

(depth information processing device)
The depth information processing device 13 according to one embodiment includes a distance information acquisition unit 41, an image acquisition unit 42, a control unit 43, a storage unit 44, and an output unit 45, as shown in FIG.

The distance information acquisition unit 41 acquires output data from the distance measuring device 11 . The distance information acquisition unit 41 may include a communication module that communicates with the distance measuring device 11 . In the case of the detection device 21 of FIG. 2 , the distance information acquisition section 41 acquires output data from the control section 24 forming part of the distance measurement device 11 . The output data includes at least two-dimensional position information indicating the detection position when the detection range is planarly viewed, and a depth measurement value that is the distance to the object ob positioned at the position. The output data may further include information on the time when the depth was detected. The time information can also be added based on the time when the depth information processing device 13 acquires the output data.

The image acquisition unit 42 acquires an image from the imaging device 12 . The image acquisition unit 42 may include a communication module that communicates with the imaging device 12 . In the case of the detection device 21 of FIG. 2 , the image acquisition section 42 acquires an image from the control section 24 forming part of the imaging device 12 . The image acquired by the image acquisition unit 42 can include information about the time when the image was captured. The time information can also be given based on the time when the depth information processing device 13 acquired the image. The time information of the output data of the distance measuring device 11 and the time information of the image must be synchronized.

The control unit 43, like the control unit 24 of the detection device 21, includes one or more processors and memory. Like the control unit 24 of the detection device 21, the control unit 43 includes at least one of a general-purpose processor that loads a specific program and executes a specific function, and a dedicated processor that specializes in specific processing. OK.

The control unit 43 controls the entire depth information processing device 13, and estimates the depth distribution at a predetermined timing based on the output data acquired by the distance information acquisition unit 41 and the image acquired by the image acquisition unit 42. various processes. The processing executed by the control unit 43 includes depth distribution estimation using machine learning, which will be described below.

The storage unit 44 includes semiconductor memory and/or magnetic memory. The storage unit 44 may function, for example, as a main memory device, an auxiliary memory device, or a cache memory. The storage unit 44 stores arbitrary information used for the operation of the depth information processing device 13 . For example, the storage unit 44 may store system programs, application programs, management databases, and the like. The storage unit 44 can temporarily store the output data acquired by the distance information acquisition unit 41, the image acquired by the image acquisition unit 42, and the information obtained by processing them.

The storage unit 44 stores a trained model 46 generated by machine learning in another computer (referred to as a “learning computer”) or the depth information processing device 13 . The trained model 46 receives the output data obtained by the distance information obtaining unit 41 and the preprocessed data of the image obtained by the image obtaining unit 42, and uses a data set that outputs the depth distribution at a predetermined timing. Generated by learning. Trained model 46 may be stored in the form of programs and parameters. The control unit 43 can read the learned model 46 from the storage unit 44 and use it.

The output unit 45 is configured to be able to output information from the depth information processing device 13 to the outside. The output unit 45 includes one or more of a display, a communication module that transmits information to an external computer, a device that outputs information to a storage medium, and the like.

(Processing for estimating depth distribution)
Depth distribution estimation processing executed by the control unit 43 will be described below with reference to the flowchart of FIG. The depth information processing device 13 may be configured to read a program recorded on a non-temporary computer-readable medium and implement processing performed by the control unit 43, which will be described below. Non-transitory computer-readable media include, but are not limited to, magnetic storage media, optical storage media, magneto-optical storage media, and semiconductor storage media.

First, the control unit 43 acquires output data of the distance measuring device 11 through the distance information acquisition unit 41, and acquires an image captured by the imaging device 12 through the image acquisition unit 42 (step S101).

FIG. 5 is a diagram showing an example of a scene within the detection range of the depth distribution detection system 10. As shown in FIG. For example, it is assumed that the distance measuring device 11 and the imaging device 12 or the detection device 21 are fixed to a structure beside or above the road and detect a moving object MO traveling on the road. A moving object MO may be, for example, an automobile, a bicycle, or a pedestrian. Although FIG. 5 includes one moving object MO, the number of moving objects MO is not limited to one. In the detection range, the portion other than the moving object MO becomes the background BG that does not change with time.

The frame rate and resolution of the ranging device 11 are lower than those of the imaging device 12 . A period of one frame of the distance measuring device 11 is defined as a first period _T1 . For example, the frame rate of the ranging device 11 is 3 fps and the first period _T1 is 1/3 seconds. Also, the number of depths that can be measured by the distance measuring device 11 during one frame can be, for example, 2400 in total, 120 in the horizontal direction and 20 in the vertical direction. For example, when the moving object MO is an automobile, the moving object MO moves within the field of view where the rangefinder 11 performs measurement while the rangefinder 11 scans one frame. Therefore, even if the depth distribution is generated by collecting the depths obtained during the first period _T1 , an instantaneous depth distribution at a specific time cannot be obtained due to the difference in measurement time. Therefore, the control unit 43 performs the following processing.

As shown in the lower part of FIG. 6, based on the output data of the distance measuring device 11, the control unit 43 calculates a plurality of depths for each of a plurality of second periods _T2 obtained by dividing the first period _T1 . It is extracted in association with information (step S102). The control unit 43 can, for example, divide the first period _T1 into 100 second periods _T2 . For example, if the length of the first period _T1 is 1/3 seconds, the length of the second period _T2 can be 1/300 seconds. The number n for dividing the first period _T1 into the second period _T2 is not limited to 100 and can be any number.

As shown in FIG. 6, for each second period _T2 , the depth detected by the distance measuring device 11 in each second period _T2 is extracted as depth information di. The depth information di is information that associates a depth measurement value with two-dimensional position information and time information indicating the depth detection position. Assume that a set of a plurality of pieces of depth information di detected in each second period _T2 is a short-term depth information distribution SDk (k=1 to n). For example, when the distance measuring device 11 measures a total of 2400 depths, 120 in the horizontal direction and 20 in the vertical direction, during one frame, and n=100, the number of pieces of depth information di included in each short-term depth information distribution SDk can be 24 in total, 12 in the horizontal direction and 2 in the vertical direction. Since FIG. 6 is a diagram for illustration only, the number of pieces of depth information di in each short-term depth information distribution SDk is indicated by a small number of dots.

In FIG. 6, the distance measuring device 11 sequentially acquires the depth of random positions within the detection range. However, the range finder 11 may acquire the depth by scanning regularly within the detection range.

Further, the control unit 43 may generate information obtained by accumulating the depth information di of one frame over the first period _T1 as the accumulated depth information AD. The cumulative depth information AD is information that becomes an image with distortion and deviation due to the difference in measurement time when the depth information di is associated with the two-dimensional coordinates of the detection range and expressed as an image.

Depth information di included in each short-term depth information distribution SDk (k=1 to n) is provided with time information corresponding to each short-term depth information distribution SDk (k=1 to n). As an example, the depth information di belonging to each short-term depth information distribution SDk (k=1 to n) may be provided with the end time of the second period _T2 as time information. For example, when the start time of the first period T ₁ is 0, the depth information di included in the short-term depth information distribution SDk (k=1 to n) contains time information of t=T ₁ ×k/n. is given. If the output data output by the distance measuring device 11 includes information on the time when the depth was measured, this may be used as the time information of the depth information di as it is. In this way, the depth measurement value included in each piece of depth information di is associated with a three-dimensional voxel containing two-dimensional coordinates indicating the detection position when the detection range is planarly viewed and time information. be done.

Next, for each second period _T2 , based on the image acquired from the imaging device 12, the control unit 43 divides the area division information SIk ( k=1 to n) (see FIG. 7) (step S103). Therefore, the control unit 43 extracts an image from the image acquired from the imaging device 12 at a predetermined timing during the first period _T1 . For example, as shown in FIG. 6, when the start time of the first period _T1 is t=0, the control unit 43 controls the image IM1 at t=0 and the end time of the first period _T1 . and an image IM2 at t= _T1 where .

The control unit 43 estimates the position of the moving object MO at the end time of each second period _T2 using the optical flow method for the time-series images acquired from the imaging device 12 . That is, as shown in FIG. 7, the control unit 43 extracts the moving object MO from the first image MI1, and extracts the same moving object MO (referred to as MO' in FIG. 7) from the second image MI2. The control unit 43 extracts the feature points of the moving object MO and calculates the displacement vector v of the feature points between the first image MI1 and the second image MI2. Assuming that each feature point moves at a constant speed, the control unit 43 estimates the position of the moving object MO at the time corresponding to each second period _T2 .

The control unit 43 determines region division information SIk (k=1 to n) based on the position of the moving object MO in each second period _T2 . The area division information SIk (k=1 to n) is area boundary AB information indicating a boundary between a moving object area MA in which a moving object MO exists and a background area BA that is an area of a background BG other than the moving object area MA. including. In FIG. 7, the area boundary AB is indicated by a rectangle for the sake of explanation only, but the area boundary is assumed to follow the outer shape of the moving object MO in plan view.

Note that the method of generating the region division information SIk (k=1 to n) is not limited to using optical flow. For example, the control unit 43 acquires an image captured at time t= _T1 ·k/n corresponding to each second period _T2 , and obtains a difference from the image in the immediately preceding second period _T2 . can be done. The control unit 43 can determine that the area where the difference is substantially 0 is the background area BA because the image has not changed. Also, the control unit 43 can determine that an area where the difference is substantially other than 0 is the moving object area MA because the image changes. Note that the fact that the difference is substantially 0 means that the difference is 0 or a small value within the detection error range.

In addition, the control unit 43 can generate a luminance histogram from a plurality of images for each pixel, and generate region division information SIk (k=1 to n) based on this histogram. A histogram is represented as a graph with luminance on the horizontal axis and frequency on the vertical axis. Based on the plurality of images, it can be determined that the position of the pixel where the frequency of specific luminance is higher than a predetermined value corresponds to the background area BA. The position of the pixel where the luminance frequency is widely distributed can be determined as the moving object area MA.

Furthermore, as shown in FIG. 8, the control unit 43 includes the accumulated depth information AD in the background area BA for each of the second periods _T2 , and sets the plurality of depth information di for the second periods _T2 to the moving object area. Synthetic depth information CDk (k=1 to n) to be included in MA is generated (step S104).

That is, the control unit 43 applies the area division information SIk (k=1 to n) generated in step S103 for each of the second periods _T2 , and the detection range of the distance measuring device 11 is two-dimensionally Split a region. The control unit 43 adopts the depth information di of the short-term depth information distribution SDk (k=1 to n) for the moving object area MA, and the depth information di of the cumulative depth information AD for the background area BA, Synthetic depth information CDk (k=1 to n) is generated. As a result, the background area BA of the composite depth information CDk (k=1 to n) contains high-density depth information di reflecting the depth information di of one frame. In the background area BA, since the object to be detected does not change over time, a large amount of depth information di accumulated over one frame can be used. On the other hand, in the moving object area MA, since the depth information di changes with time, low-resolution depth information di for each second period _T2 is used. Note that FIG. 8 is a diagram for illustration only, and the depth information di is represented by a small number of points. Depth information di can be obtained at a higher density.

By increasing the density of the depth information di of the background area BA in this way, it is possible to improve the estimation accuracy of the depth distribution when using the composite depth information CDk (k=1 to n) as input for machine learning. can be done.

Next, the control unit 43 inputs the n composite depth information CDk (k=1 to n) generated in step S104 to the trained model 46, and estimates the depth distribution at a predetermined timing (step S105). . The predetermined timing can be set at any time during the first period _T1 . For example, the predetermined timing can be set to approximately the middle time within the first period _T1 . That is, when the start time of the first period _T1 is t=0, the predetermined timing can be near t= _T1 /2. For example, if the predetermined timing is equal to or greater than T ₁ ×0.4 and equal to or less than T ₁ ×0.6, the predetermined timing can be said to be approximately the central time within the first period T ₁ . The control unit 43 can store the estimated depth distribution at the predetermined timing in the storage unit 44 . Further, the control unit 43 can output the estimated depth distribution at a predetermined timing from the output unit 45 .

(learning computer)
The trained model 46 used for estimating the depth distribution of the depth information processing device 13 is generated in advance by the learning computer 50 . The learning computer 50 can be a computer different from the depth information processing device 13 . Alternatively, the depth information processing device 13 may have the function of the learning computer 50 . In one embodiment, the learning computer 50 generates teacher data for machine learning through simulation, and uses this teacher data to generate the trained model 46 .

The learning computer 50 moves a virtual moving object MO within a virtual three-dimensional space, and outputs data and images detected by the distance measuring device 11 and the imaging device 12 placed at predetermined positions in the virtual space. calculate. The learning computer 50 can generate images by CG (Computer Graphics). Further, the learning computer 50 calculates a depth distribution without delay at a predetermined timing by simulation calculation.

The learning computer 50 includes, for example, an input unit 51, a calculation unit 52, a storage unit 53, and an output unit 54, as shown in FIG.

The input unit 51 receives input of various data used for simulation. The input unit 51 includes an input device such as a keyboard and a mouse, a communication module, and/or a reading device for a storage medium. For example, the input unit 51 receives input of information such as the position and size of a stationary object or background placed in the virtual space. The input unit 51 also receives input of the position, size, moving direction and speed of the moving object MO placed in the virtual space. Further, the input unit 51 receives input of information on the positions and orientations of the distance measuring device 11 and the imaging device 12 placed in the virtual space. Furthermore, the input unit 51 may receive settings of various conditions for performing the simulation. For example, the input unit 51 receives settings for the first period _T1 and the second period _T2 .

The calculation unit 52 executes various calculations for generating the trained model 46 . The calculation unit 52 includes one or more processors, like the control unit 24 of the detection device 21 and the control unit 43 of the depth information processing device 13 . The calculation unit 52 may include a simulation unit 55 that performs simulation and a function approximator 56 that performs machine learning. The function approximator 56 estimates the relationship between inputs and outputs from teacher data, which is a data set in which a plurality of inputs and outputs are combined. Function approximator 56 is, for example, a neural network. In this embodiment, simulation and machine learning are performed by the same learning computer 50, but they may be performed by separate computers.

The storage unit 53 includes semiconductor memory and/or magnetic memory. The storage unit 53 can store arbitrary information used for the operation of the learning computer 50 and arbitrary information output by the learning computer 50 . For example, the storage unit 53 can store various setting information of the simulation acquired from the input unit 51 and the learned model 46 generated as a result of machine learning.

The output unit 54 is configured to be able to output information from the learning computer 50 to the outside. The output unit 54 includes a display, a communication module that transmits information to an external computer, a device that outputs information to a storage medium, and the like.

(Generation of trained model)
The generation processing of the learned model 46 executed by the calculation unit 52 will be described below with reference to the flowchart of FIG. 10 . The learning computer 50 may be configured to read a program recorded on a non-temporary computer-readable medium and implement part or all of the processing performed by the computing unit 52 described below. Non-transitory computer-readable media include, but are not limited to, magnetic storage media, optical storage media, magneto-optical storage media, and semiconductor storage media.

The computing unit 52 generates the output data of the distance measuring device 11 in the virtual space, the image output from the imaging device 12, and the depth distribution at a predetermined timing of the first period _T1 by simulation. Hereinafter, the depth distribution at a predetermined timing is called a target depth distribution. The target depth distribution is a correct depth distribution at a predetermined timing that does not include deviations due to differences in measurement times within one frame of the rangefinder 11 . The calculation unit 52 generates output data according to the frequency at which the distance measuring device 11 can actually measure the depth. The calculation unit 52 generates a target depth distribution including a larger number of pieces of depth information di than the number of pieces of depth information di that can actually be detected by the distance measuring device 11 in the first period _T1 , which is a period of one frame. good. For example, when the distance measuring device 11 acquires depth information di of detection positions arranged horizontally 120 times vertically during the first period _T1 , the target depth distribution calculated by simulation is It can be a distribution of depths of detection positions arranged 240 in the direction and 40 in the vertical direction. That is, the target depth distribution can include depth information di with a higher resolution than depth information di that the distance measuring device 11 can actually detect.

Next, the calculation unit 52 converts the output data of the distance measuring device 11 obtained as a result of the simulation into a plurality of depth information di as time information for each of the second periods _T2 obtained by dividing the first period _T1 . It is associated and extracted (step S202). This process is the same process as step S102 in the flowchart of FIG. The first period _T1 and the second period _T2 used here are set to be the same as the first period _T1 and _the second period T2 used in step S102.

Based on the image obtained as a result of the simulation, the calculation unit ₅₂ generates area division information SIk (k=1 ˜n) are generated (step S203). This process is the same process as step S103 in the flowchart of FIG.

Furthermore, for each of the second periods _T2 , the computing unit 52 includes, in the background area BA, cumulative depth information AD obtained by accumulating a plurality of pieces of depth information di in the first period _T1 . Synthetic depth information CDk (k=1 to n) including a plurality of pieces of depth information di of _T2 in the moving object area MA is generated (step S204). This process is the same process as step S104 in the flowchart of FIG.

That is, the calculation unit 52 performs the same processing as the processing performed by the depth information processing device 13 in the pre-stage of inference by machine learning on the output data of the distance measuring device 11 and the image of the imaging device 12 obtained by the simulation. to generate synthetic depth information CDk (k=1 to n).

The calculation unit 52 treats a combination of a plurality of synthetic depth information CDk (k=1 to n) corresponding to one frame of data and the target depth distribution as one data, and generates teacher data including a plurality of such data (step S205). ). The calculation unit 52 may generate a plurality of data in parallel in steps S201 to S204. The computing unit 52 generates a large number of data sets, for example, 1000 to 10000 sets of data for machine learning.

The computing unit 52 uses the function approximator 56 to perform machine learning using the composite depth information CDk (k=1 to n) as an input and the target depth distribution as an output, thereby generating a trained model 46 (step S206). . The calculation unit 52 stores the trained model 46 in the storage unit 53 and/or outputs it from the output unit 54 . The trained model 46 can be used by multiple depth information processing devices 13 .

(Calculation method of machine learning)
Function approximator 56 is, for example, a neural network. network
is a depth estimate for the input, as shown in equation (1)
and uncertainty
and

Uncertainty represents the variance in calculating the output of machine learning as a probability representation of a normal distribution. The function approximator 56 performs calculations for calculating the loss function L(θ) of Equation (2), and performs learning so as to minimize this loss function L(θ).

Here, i indicates each data included in the teacher data. Also, Di is the depth of the teacher data. The uncertainty used in Equation (2) represents the extent of data that are difficult to estimate.

The second term of Equation (2) acts to reduce the uncertainty (variance). On the other hand, the exponential function exp() of the first term acts to increase the uncertainty (variance). Since the calculation unit 52 performs learning so that the loss function L(θ) becomes small, learning can be performed while balancing the first term and the second term. That is, according to the present embodiment, uncertainty is taken into account in generating the trained model 46 . By learning using uncertainty, we can detect depth measurements that are spatially distant and near object boundaries, temporally distant from the target time, and depth measurements that are difficult to estimate such as moving objects. Therefore, it can be expected that the output will be robust.

In the present embodiment, as described above, the learning computer 50 provides the depth information di of each second period _T2 with time information of higher resolution than that of the first period _T1 for machine learning. generated composite depth information CDk (k=1 to n) as an input for . In addition, the learning computer 50 generates a target depth distribution that does not include distortion based on the deviation of the measurement time by simulation, and uses it as an output of machine learning. Accordingly, by using the generated trained model 46, the depth information processing device 13 can generate composite depth information CDk (k= 1 to n) are input to the learned model 46 to estimate the depth distribution at a predetermined timing. The estimated depth distribution is obtained by reducing the distortion and deviation of the depth distribution caused by the deviation of the measurement time of the distance measuring device 11 .

Further, in the present embodiment, in both stages of learning by the learning computer 50 and estimation by the depth information processing device 13, the depth information di is accumulated in the background area BA over the first period _T1 as an input for machine learning. The cumulative depth information AD obtained by As a result, it is possible to increase the density of the input information more than simply using the short-term depth information distribution SDk (k=1 to n) obtained by dividing one frame of ranging information into n pieces for the input of machine learning. can. This improves the accuracy of machine learning, thereby improving the prediction accuracy of the depth distribution at a predetermined timing.

Furthermore, in this embodiment, in the learning stage of machine learning, calculations involving uncertainty are performed to generate the trained model 46 . As a result, uncertainty can be reduced even under conditions of position and time where estimation of depth information di is difficult. This makes it possible to further improve the estimation accuracy of the depth distribution.

In fact, according to an experiment conducted by the inventor of the present application by simulation, the effect of improving the estimation accuracy of the depth distribution by dividing the depth information di that is the input of machine learning in terms of time and giving it time information was confirmed. It was also confirmed that the background area BA has the depth information di accumulated for one frame to increase the density of the background area BA, thereby improving the estimation accuracy of the depth distribution. Furthermore, it was confirmed that considering uncertainty in the learning stage of machine learning also has the effect of improving the estimation accuracy of the depth distribution. It was also confirmed that the estimation accuracy of the depth distribution is further improved by executing these methods in combination.

Further, according to the present embodiment, the number of depth information di of the target depth distribution as an output used in the learning stage is output from the distance measuring device 11 as one frame of depth information di in the first period _T1. It is made larger than the number of depth information di. As a result, the depth information processing device 13 can acquire a depth distribution with a higher density than the depth distribution that the distance measuring device 11 can acquire in one frame as the depth distribution at the predetermined timing. Further, in this way, when generating a depth map based on the depth distribution, the depth information processing device 13 can generate a depth map by increasing the resolution of the depth distribution output from the distance measuring device 11 .

Although the embodiments of the present disclosure have been described with reference to the drawings and examples, it should be noted that those skilled in the art can easily make various variations or modifications based on the present disclosure. Therefore, it should be noted that these variations or modifications are included within the scope of this disclosure. For example, functions included in each component or each step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined into one or divided. is. Embodiments according to the present disclosure can also be implemented as a method, a program, or a storage medium recording a program executed by a processor provided in an apparatus. It should be understood that these are also included within the scope of the present disclosure.

REFERENCE SIGNS LIST 10 depth distribution detection system 11 distance measuring device 12 imaging device 13 depth information processing device 14 optical system 21 detection device 22 irradiation section 23 reflection section 24 control section 25 first optical system 26 detection element 27 second optical system 28 imaging element 29 pre-stage optical system 30 first post-stage optical system 31 second post-stage optical system 32 switching section 41 distance information acquisition section 42 image acquisition section 43 control section 44 storage section 45 trained model 50 learning computer 51 input section 52 calculation section 53 storage unit 54 output unit 55 simulation unit 56 function approximator ax optical axis MO moving object BG background T ₁ first period T ₂ second period IM1 first image IM2 second image di depth information SDk (k=1 ~ n) Short-term depth information distribution AD Cumulative depth information SIk (k = 1 to n) Region division information CDk (k = 1 to n) Composite depth information

Claims (11)

a distance information acquisition unit that acquires output data from a scanning rangefinder;
an image acquisition unit that acquires an image of a range that overlaps the detection range of the distance measuring device;
A processor for estimating a depth distribution at a predetermined timing within a first period, which is a period of one frame of the range finder, based on the output data and the image,
The processor
From the output data, for each of a plurality of second periods obtained by temporally dividing the first period, the depth, the position in the two-dimensional direction where the depth was detected, and time information corresponding to the second period are obtained. Generate multiple depth information containing,
generating area division information for dividing the detection range into a background area and a moving object area for each of the second periods from the image;
For each of the second periods, the background area includes accumulated depth information obtained by accumulating the plurality of depth information in the first period, and the plurality of depth information in the second period is included in the generate synthetic depth information, including in the region of the moving object;
The synthesized depth information of a plurality of the second periods is input, and the generated synthesis is applied to the trained model trained using teacher data whose output is the depth distribution at the predetermined timing within the first period. A depth information processing apparatus comprising: a processor that estimates the depth distribution by inputting depth information. The trained model is generated by performing an operation including uncertainty when performing learning using the teacher data, and the uncertainty is calculated as a probability expression of normal distribution based on the output of machine learning. 2. The depth information processing apparatus according to claim 1, representing a variance in the case of . 3. The depth information processing apparatus according to claim 1, wherein the optical axis of the optical system of said distance measuring device and the optical axis of the optical system for capturing said image are substantially aligned. The segmentation information is generated from the images based on an optical flow, a difference between the images, or a luminance histogram obtained from a plurality of the images, for the images in time series. 4. The depth information processing device according to any one of 1 to 3. 5. The depth information processing apparatus according to any one of claims 1 to 4, wherein said predetermined timing is a substantially central time within said first period. The depth information processing apparatus according to any one of claims 1 to 5, wherein the estimated depth distribution includes a number of pieces of depth information greater than a total number of pieces of depth information generated during the first period. . 7. The depth information processing according to any one of claims 1 to 6, wherein the synthetic depth information and the depth distribution of the teacher data are generated by executing a simulation that virtually places a moving object in the detection range. Device. A depth distribution estimation method executed by a computer processor,
Acquire output data from a scanning rangefinder,
Acquiring an image of a range overlapping the detection range of the rangefinder,
From the output data, the depth, the position in the two-dimensional direction where the depth was detected, and the generating a plurality of depth information including time information corresponding to the second period;
generating area division information for dividing the detection range into a background area and a moving object area for each of the second periods from the image;
For each of the second periods, the background area includes accumulated depth information obtained by accumulating the plurality of depth information in the first period, and the plurality of depth information in the second period is included in the generate synthetic depth information, including in the region of the moving object;
The generated synthetic depth is added to a trained model that has been trained using teacher data that receives a plurality of synthetic depth information of the second period as an input and outputs a depth distribution at a predetermined timing within the first period. A method of estimating the depth distribution by inputting information. a scanning rangefinder;
an imaging device that captures an image of a range that overlaps with the detection range of the distance measuring device;
Depth information including a processor that calculates a depth distribution at a predetermined timing within a first period, which is a period of one frame of the range finder, based on the output data of the range finder and the image captured by the imaging device. a processing device;
The processor
From the output data, for each of a plurality of second periods obtained by temporally dividing the first period, the depth, the position in the two-dimensional direction where the depth was detected, and time information corresponding to the second period are obtained. Generate multiple depth information containing,
generating area division information for dividing the detection range into a background area and a moving object area for each of the second periods from the image;
For each of the second periods, the background area includes accumulated depth information obtained by accumulating the plurality of depth information in the first period, and the plurality of depth information in the second period is included in the generate synthetic depth information, including in the region of the moving object;
The synthesized depth information of a plurality of the second periods is input, and the generated synthesis is applied to the trained model trained using teacher data whose output is the depth distribution at the predetermined timing within the first period. A depth distribution detection system that estimates the depth distribution by inputting depth information. A trained model generation method executed by a computer,
a plurality of pieces of depth information and region division information generated for each of a plurality of second periods obtained by temporally dividing a first period, which is a period of one frame of a scanning rangefinder; data obtained by combining depth distributions at predetermined timings within a period, wherein each of the depth information includes depth, a position in the two-dimensional direction where the depth is detected, and time information corresponding to the second period; the area division information obtains or generates a plurality of data for dividing the detection range of the rangefinder into a background area and a moving object area;
For each of the second periods, the background area includes accumulated depth information obtained by accumulating the plurality of depth information in the first period, and the plurality of depth information in the second period is included in the generate synthetic depth information, including in the region of the moving object;
generating a learned model by performing machine learning using the synthetic depth information of the plurality of the second periods as input and teacher data having the depth distribution as output;
Trained model generation method. 3. When generating said trained model using said teacher data, an operation including uncertainty is performed, and said uncertainty represents a variance in the case of calculating an output of machine learning as a probability expression of normal distribution. 11. The learned model generation method according to 10.

Images