JP2024043269A - Processor - Google Patents

Abstract

To provide a processor which can correspond to occurrence of high luminance abnormality caused by a pulse emission type optical sensor.SOLUTION: A processor 10 includes a processor 10b so that corresponding processing to an image photographed by an on-vehicle camera 1 having a predetermined shutter system is reflected on a driving action in a vehicle EV capable of executing automatic driving. The processor 10b is configured to execute: acquisition of an image; extraction of an abnormality occurrence image where high luminance abnormality occurs by incidence of a light pulse LP from a rider 80 as a pulse emission type optical sensor on the on-vehicle camera 1, from the image, using an extraction algorithm corresponding to the shutter system; and making of the corresponding processing to the abnormality occurrence image different from corresponding processing to other images.SELECTED DRAWING: Figure 4

Description

The disclosure in this specification relates to technology for performing advanced driving such as automatic driving.

Patent Document 1 discloses that a vehicle is equipped with both an on-vehicle camera and a pulse emission type optical sensor such as an optical radar. Further, a vehicle control device is disclosed that detects the occurrence of flare caused by sunlight and determines the reliability of an image taken by an on-vehicle camera based on this result.

Patent No. 6981224

Now, with the spread of autonomous driving using in-vehicle cameras and pulse-emitting optical sensors, pulses emitted by pulse-emitting optical sensors are often incident on the in-vehicle camera, causing high-intensity abnormalities such as flare. The inventor has discovered that there is a possibility. However, the device of Patent Document 1 does not take into consideration the form of occurrence of high brightness abnormalities caused by such a pulsed light emitting optical sensor.

One of the objects of the disclosure of this specification is to provide a processing device that can deal with the occurrence of high brightness abnormalities caused by pulsed light emitting optical sensors.

One of the aspects disclosed herein is that in a vehicle (EV) capable of autonomous driving, processing for images taken by an in-vehicle camera (1, 2, 401) with a predetermined shutter method is reflected in driving actions. A processing device comprising at least one processor (10b, 310b) to
At least one processor is
Obtaining an image;
From the image, use an extraction algorithm compatible with the shutter method to extract an abnormality image in which a high-intensity abnormality occurs due to the light pulse (LP) from the pulse-emitting optical sensor (3, 80) entering the in-vehicle camera. to do and
A processing device configured to perform processing for handling an abnormality image differently from handling processing for other images.

According to this aspect, a high-intensity abnormality caused by the incidence of a light pulse (LP) from a pulse-emitting optical sensor is extracted using an extraction algorithm compatible with the shutter method of the vehicle-mounted camera. Therefore, it is possible to extract high-intensity abnormalities caused by the pulse-emitting optical sensor according to the characteristics of the vehicle-mounted camera. Since the handling process for an image in which a high brightness abnormality has occurred is different from that for other images, it is possible to take appropriate measures against the occurrence of a high brightness abnormality caused by a pulse emission type optical sensor.

Note that the reference numerals in parentheses included in the claims etc. exemplarily indicate correspondence with parts of the embodiment described later, and are not intended to limit the technical scope.

A diagram showing a schematic configuration of an automatic driving system. A diagram showing the functional configuration of an automatic driving system. The figure which shows the example of the scene where the light pulse of the lidar of another vehicle is incident on the vehicle-mounted camera of the own vehicle. FIG. 3 is a diagram illustrating exposure using a rolling shutter method. A diagram showing an example of an abnormality occurrence image. FIG. 13 is a diagram showing an example of a normal image. A graph showing an example of a brightness value map when a high brightness abnormality occurs. FIG. 3 is a diagram for explaining the relationship between a change in exposure time and the number of lines in which a high-luminance abnormality occurs. Flowchart showing an example of a processing method. FIG. 3 is a diagram illustrating exposure using a global shutter method. The figure which shows the example of the scene where the light pulse of the lidar of a host vehicle enters the vehicle-mounted camera of a host vehicle. A diagram showing the functional configuration of an automatic driving system. Flowchart showing an example of a processing method. A diagram showing a schematic configuration of an automatic driving system. The schematic diagram which shows the vehicle-mounted camera provided with a polarizing filter. A diagram for explaining the function of a polarizing filter.

Hereinafter, a plurality of embodiments will be described based on the drawings. Note that redundant explanation may be omitted by assigning the same reference numerals to corresponding components in each embodiment. When only a part of the configuration is described in each embodiment, the configuration of the other embodiments previously described can be applied to other parts of the configuration. Furthermore, in addition to the combinations of configurations specified in the description of each embodiment, it is also possible to partially combine configurations of multiple embodiments even if not explicitly specified, as long as there is no particular problem with the combination. .

(First embodiment)
The processing device 10 of the first embodiment shown in FIG. 1 is an electronic control unit (ECU) mounted on a vehicle EV. This vehicle EV may mean the own vehicle or the host vehicle. The processing device 10 is communicably connected to an on-vehicle camera 1 and a lidar (LiDAR, Light Detection and Ranging/Laser imaging Detection and Ranging) 3 that are also mounted on the vehicle EV. Specifically, the processing device 10 is connected to the in-vehicle camera 1 and the lidar 3 using at least one type of wired communication means such as a wire harness and wireless communication means such as CAN (registered trademark) as an in-vehicle network. Ru. The processing device 10 may constitute an automatic driving system together with the in-vehicle camera 1 and the lidar 3.

Here, the front, front side, side, rear side, rear, horizontal direction, and vertical direction used in the following description may be defined with the vehicle EV on a horizontal plane as a reference.

The vehicle-mounted camera 1 is configured to photograph the outside world of the vehicle EV. The vehicle-mounted camera 1 may be arranged to photograph a predetermined range of the outside of the vehicle EV, such as the front, front side, side, rear side, and rear of the vehicle EV. A plurality of in-vehicle cameras 1 may be mounted on one vehicle EV. In this case, the plurality of vehicle-mounted cameras 1 may be arranged so as to photograph mutually different ranges of the outside world of the vehicle EV.

The vehicle-mounted camera 1 referred to here may be an independent module with a photographing function unit called a camera head. For example, the vehicle-mounted camera 1 includes an optical system, an imaging device, and a photographing circuit. The optical system includes, for example, one or more lenses, and collects incident light from the photographing area and forms an image on the imaging element. The imaging element is, for example, a CCD sensor or a CMOS sensor, and can output the light detection result for each pixel arranged two-dimensionally along a first direction and a second direction that are orthogonal to each other. For example, the first direction is set along the horizontal direction of the vehicle EV, and the second direction is set along the vertical direction of the vehicle EV.

The photographing circuit electrically controls the shutter timing of the image sensor and acquires detection data of each pixel of the image sensor. The imaging circuit may further generate image data that can be output as a video signal to an external device. That is, the photographing circuit may convert the detection data into the form of image data and output it to an external device.

The vehicle-mounted camera 1 of this embodiment employs a rolling shutter method as a shutter method. In this method, the imaging circuit reads the detection results of each pixel of each pixel of the image sensor for each line along the first direction. When the imaging circuit completes reading out the detection results for one line, it reads out the next line shifted by one line in the second direction. When reading of all lines constituting an image is completed, one frame's worth of image data can be generated.

In other words, the timing of starting exposure is shifted for each line. For this reason, the rolling shutter method is characterized by the fact that even within the same frame, there is a shift in the photographing timing for each line, that is, there is a time difference in the photographing time.

The lidar 3 is a pulse emission type optical sensor that emits a light pulse LP and detects an object using the reflected light. Further, the lidar 3 is a pulse scanning optical sensor that scans the measurement area with a light pulse LP and detects an object present in the measurement area using the reflected light. Furthermore, the lidar 3 is also a distance sensor that measures the distance to an object using, for example, a ToF (Time of Flight) method.

The lidar 3 scans the measurement area with a light pulse LP projected toward the measurement area, and receives reflected light reflected by an object. This allows the rider 3 to detect objects existing in the measurement area and their distances.

For example, the lidar 3 has a configuration including a light emitting section, a light receiving section, and a control circuit. The light emitting section has a light source that emits light pulses LP toward the measurement area. One or more light sources may be provided. The light source may be, for example, a laser diode or an LED as long as it can emit light in a pulsed manner. The light pulse LP emitted by the light source may be linearly polarized or non-polarized. The light emitting unit may further include an optical system that collects the light pulse LP from the light source. The scanning of the light pulse LP may be realized by changing the projection angle using an optical element such as a mirror or a lens, or may be realized by controlling the phase and interference state of light in the light emitting section.

The light receiving section includes a light receiving element that receives reflected light from the light pulse LP that is reflected by an object in the measurement area and returns. As the light receiving element, for example, an APD (Avalanche Photo Diode) or a SPAD (Single Photon Avalanche Diode), which is a type of APD, can be used. The light receiving section may further include an optical system that collects the reflected light. Note that there is a high possibility that the reflected light is also pulsed.

The control circuit controls the operation of the rider 3. Specifically, the control circuit controls the light emission timing of the light source and the scanning direction so as to be linked to each other.

The processing device 10 is realized mainly using a computer, for example. A computer implementing the processing device 10 may include at least one memory 10a and at least one processor 10b. The memory 10a is at least one type of non-transitional physical storage medium, such as a semiconductor memory, a magnetic medium, an optical medium, etc., that non-temporarily stores programs, data, etc. that can be read by the processor 10b. good. Furthermore, a rewritable volatile storage medium such as a RAM (Random Access Memory) may be provided as the memory 10a. The processor 10b includes, as a core, at least one type of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU.

Further, the computer may be an SoC (System on a Chip) in which the memory 10a, the processor 10b, and an interface are integrated into one chip, or may have an SoC as a component of the computer. Further, the processing device 10 may include an FPGA (Field-Programmable Gate Array) as an element of a computer or as an element separate from the computer. The processor 10b may control the FPGA to perform simple repetitive processing in image processing, thereby speeding up the processing.

Next, the functions of the processing device 10 will be described in detail using FIG. 2. The processing device 10 has a configuration including an image acquisition section 51, a flare extraction section 52, a correspondence processing section 53, an object recognition section 54, and a driving planning section 55 as functional blocks realized by a processor 10b that executes a program.

The image acquisition unit 51 acquires images captured by the vehicle-mounted camera 1. Specifically, the image acquisition unit 51 may acquire image data generated by the vehicle-mounted camera 1. Alternatively, the processing device 10 may acquire detection data by the vehicle-mounted camera 1 from the vehicle-mounted camera 1, and the image acquisition unit 51 may acquire image data generated based on this data.

The image acquisition unit 51 may further specify information about the vehicle-mounted camera 1 that captured the image. The information on the vehicle-mounted camera 1 may include camera specifications including the shutter method, photographing conditions including the exposure time (also referred to as shutter speed), and the like. The image data acquired by the image acquisition section 51 is provided to the flare extraction section 52.

The flare extraction unit 52 detects, in the image acquired by the image acquisition unit 51, that a light pulse LP from a pulse-emitting optical sensor (for example, a lidar 80) of another vehicle OV is incident on the in-vehicle camera 1, as shown in FIG. 3, for example. By this, it is determined whether a high brightness abnormality (for example, flare) is occurring. In other words, the flare extraction unit 52 extracts an abnormality image in which flare caused by the optical sensor has occurred. The flare extraction section 52 corresponds to an abnormality extraction section.

An extraction algorithm corresponding to the shutter method of the vehicle-mounted camera 1 that took the image is used to extract the abnormality image. The extraction algorithm is stored as a program in the memory 10a. The extraction algorithm here may be configured to be able to detect flare that occurs in images taken using the shutter method. That is, the extraction algorithm may be universally applicable to other shutter methods, and does not need to be specialized only for that shutter method.

Here, the characteristics of an abnormality occurrence image using the rolling shutter method will be explained. As shown in FIG. 4, in the rolling shutter method, a time difference occurs in the photographing time of one frame of an image. This time difference is usually larger than the light emission duration of the light pulse LP of the lidar 3. Therefore, flare occurs only in a portion of one frame of the image. More specifically, as shown in FIG. 5, flare occurs only in a band-shaped portion corresponding to a part of the line HBL along the horizontal direction of the pixels, for example.

For further understanding, the example shown in FIG. 4 will be explained. The upper part of FIG. 4 shows the relationship between the rider 80 of the other vehicle OV and the on-vehicle camera 1. The lower part of FIG. 4 shows the exposure when the in-vehicle camera 1 shoots one frame, and the exposure is sequentially changed from the upper line to the lower line in the horizontal direction with a time difference. The manner in which it is carried out is illustrated. It is assumed that the exposure time of each line is ET1. Here, it is assumed that one frame of exposure is started at time t0. A time difference Δt occurs from the start of exposure (t0) for the top line to the start of exposure for the bottom line. Time t2 is the time when one frame of exposure is completed, and t2=t0+Δt.

As shown in the upper part of FIG. 4, it is assumed that the light pulse LP of the lidar 3 is scanned, and the light pulse LP enters the on-vehicle camera 1 at, for example, time t1. Here, it is assumed that t0<t1<t0+Δt. In this case, only the central line HBL in the vertical direction of the image is exposed when the light pulse is incident. In this case, for example, if FIG. 5 is taken in darkness such as at night, the area corresponding to the line HBL in the center will have extremely high brightness, while the other areas will be a low brightness image taken in darkness. In this way, significant brightness differences occur depending on the area.

Furthermore, when the frames before and after the frame in FIG. 5 are photographed when no light pulse is incident, as shown in FIG. 6, there is no high-luminance area and almost the entire frame is a low-luminance area.

In FIGS. 5 and 6, dot hatching is applied to high-intensity areas where flare occurs, and no hatching is applied to low-intensity areas where flare does not occur.

The extraction algorithm compatible with the rolling shutter method employs one of the following three extraction methods: method 1, method 2, or method 3.

The first method is to extract images in which an abnormality has occurred based on time-series data of images including a plurality of consecutive frames. For example, the flare extraction unit 52 may compare the frame to be inspected and the frame immediately before it. The flare extraction unit 52 may further add a plurality of frames before the inspection target to comparison targets, or may add one or more frames after the inspection target to comparison targets. Specifically, when the difference in characteristics between the frames to be compared is large, the flare extraction unit 52 determines that flare has occurred.

For example, the flare extraction unit 52 may determine that flare has occurred if a brightness value histogram obtained by converting brightness values into a histogram changes significantly between frames. A large change between frames may be a change that satisfies preset flare occurrence determination conditions. The flare extraction unit 52 may determine a change in color instead of the brightness value.

Furthermore, the flare extraction unit 52 may use a particle filter to detect the occurrence of flare. That is, in the above method, the frame to be inspected and the frames before and after are directly compared, but the flare extraction unit 52 estimates the frame to be inspected from the frames before and after the frame to be inspected using a particle filter or the like. An image may be identified and compared to the actual object under examination.

The second method is to extract, from a brightness value map in the vertical direction perpendicular to the line, a step shape whose value differs only in a portion corresponding to the above-mentioned band-shaped portion. As shown in the map in FIG. 7, the flare extraction unit 52 calculates the integrated value or average value of the luminance values of pixels on the same line, that is, multiple pixels in the horizontal direction, and Create a map of In this map, the flare extraction unit 52 detects a step shape in which only a part of the lines has a high integrated value or average value, and identifies the occurrence of flare according to the confirmation of the step shape.

Note that in the extraction algorithm, the first method and the second method may be used together. In this case, the flare extraction unit 52 may conclude that the image is an abnormal image only when it is determined that flare has occurred using both the first method and the second method.

In the third method, after detecting the step shape using the same method as the second method, the flare extraction unit 52 transmits a request to change the exposure time to the vehicle-mounted camera 1. When the image acquisition unit 51 acquires an image whose exposure time has been changed in response to the change request, the flare extraction unit 52 detects the step shape of the image using a method similar to the second method. Here, when the exposure time is changed in the rolling shutter method, the line HBL where flare occurs due to the light pulse LP changes depending on the exposure time (also refer to the explanation of the sixth corresponding process described later). When the flare extraction unit 52 detects a change in the width of the stepped shape according to the exposure time, it determines that flare has occurred.

The correspondence processing section 53 determines correspondence in image processing according to the extraction result by the flare extraction section 52. The response processing unit 53 determines that the image is reliable for normal images that are not abnormal images, and executes a response process that provides the image as it is to the object recognition unit 54 without performing any special response processing. do. The response processing unit 53 performs special response processing on the abnormality image based on the assumption that flare has occurred.

The response processing unit 53 executes one of the following response processes: the first response process, the second response process, the third response process, the fourth response process, the fifth response process, and the sixth response process. This response process is not usually determined each time, but may be set in advance as one type of response, for example, at the product design stage. On the other hand, the response process may be selected each time according to preset conditions.

The first response process is a process of deleting all frames corresponding to images in which an abnormality has occurred so that their use for object recognition or automatic driving is restricted. The response processing unit 53 may notify the object recognition unit 54 or the driving planning unit 55 that the abnormality image has been deleted.

The second correspondence process is a process in which only a portion of the image in which an abnormality occurs has a high luminance value and is transplanted from the corresponding pixel of the previous frame. Transplanting may mean replacing images. The portion with a high luminance value may be a line where flare occurs or a portion where a step shape occurs among all lines scanned by the global shutter method. If transplantation is difficult, the abnormality image itself may be deleted as in the first corresponding process.

The third response process is a process of correcting the brightness of the flare-occurring parts (i.e., high-brightness abnormal parts) in the image where an abnormality has occurred so that they become darker. For example, by correcting the high-brightness abnormal parts to the same brightness as other parts, the corrected image can be used for object recognition or autonomous driving in the same way as a normal image.

The fourth corresponding process is a process of masking only the portions with high brightness values in the image where the abnormality has occurred. Masking here means excluding from objects used in object recognition or automatic driving.

The fifth correspondence process is a process in which images are given reliability and used for object recognition or automatic driving. The fifth response process is a process that lowers the reliability of a high-luminance abnormality part of the abnormality image than the reliability of other parts.

The sixth response process is a process of outputting a request to the vehicle-mounted camera 1 to change the exposure time from the exposure time adopted as the shooting condition for the current abnormality occurrence image to the exposure time for the next shooting onwards. For example, by changing the exposure time ET1 shown in FIG. 4 to the shorter exposure time ET2 shown in FIG. 8, the number of lines HBL exposed during the time when the light pulse LP is incident decreases, and therefore the range of influence of the flare in one frame image may become smaller.

Furthermore, the response process may be a process that combines the sixth response process with one of the first to fifth response processes.

After executing the above-described handling process, the correspondence processing unit 53 provides the abnormality occurrence image to the object recognition unit 54, except when all abnormality occurrence images are deleted. Even when all abnormality images have been deleted, the response processing unit 53 may notify the object recognition unit 54 that the image cannot be provided because the image has been deleted.

The object recognition unit 54 recognizes an object reflected in the provided image of the in-vehicle camera 1. Object recognition is performed using, for example, an object recognition algorithm. The object recognition algorithm is stored as a program in the memory 10a. The object recognition algorithm may be an object recognition model that is a trained model, including, for example, a neural network. The object recognition model uses, for example, a semantic segmentation method to recognize and output objects reflected in an image.

Here, the object recognition unit 54 may refer to the reliability given to the image when recognizing the object. The object recognition unit 54 may change the object recognition judgment and its result depending on the reliability given to the image. Alternatively, the object recognition unit 54 may set the reliability given to the object according to the reliability given to the image. For example, if the reliability given to an image is low, the reliability in recognition of objects reflected in the image may also be set low.

When the reliability of the image partially decreases in the fifth correspondence process, the object recognition unit 54 may recognize the object by giving importance to information on the portion other than the portion where the reliability decreases. If the reliability of the image is partially reduced in the fifth corresponding process, the object recognition unit 54 may set the reliability to be given to the object reflected in that part to be low.

The object recognition unit 54 may further recognize objects from the detection results of the rider 3. The object recognition unit 54 may fuse the information of the vehicle-mounted camera 1 and the information of the rider 3 to improve object recognition accuracy. Information about the object recognized by the object recognition unit 54 is provided to the driving planning unit 55.

The driving planning unit 55 uses the information about the object recognized by the object recognition unit 54 to plan automatic driving and derive driving actions for automatic driving. Since the object information obtained by the object recognition section 54 is based on the response processing performed by the response processing section 53, it can be said that the driving planning section 55 essentially derives driving actions based on the results of the response processing section 53. Specifically, the driving planning unit 55 plans a traveling trajectory and a traveling speed of the vehicle EV to avoid collision with an object. Here, the driving planning unit 55 may plan automatic driving with reference to the reliability given to the object recognized by the object recognizing unit 54.

Based on the driving action derived by the driving planning unit 55, the power train, brake actuator, steering system, etc. of the vehicle EV are driven, so that the vehicle EV is automatically driven. Note that the term "automated driving" here may mean, for example, level 3 or higher automated driving defined in SAE J3016. For example, level 3 indicates that driving is conditionally automated. Level 4 indicates that driving is highly automated. Level 5 indicates that driving is fully automated.

Next, an example of a processing method by the processing device 10 for extracting images in which an abnormality has occurred and taking appropriate action according to the extraction results will be described with reference to the flowchart in FIG. 9. The series of processes shown in steps S11 to S15 is executed, for example, each time the vehicle-mounted camera 1 generates a frame of image, or each time the processing device 10 starts updating the autonomous driving plan.

In S11, the image acquisition unit 51 acquires the latest image (for example, one frame) from the vehicle-mounted camera 1. After processing in S11, the process advances to S12.

In S12, the flare extraction unit 52 determines whether flare has occurred in the image acquired in S11. Specifically, it is determined whether there is a significant change in the brightness value based on the difference in the time series data (whether the brightness value has changed by more than a preset threshold). If Yes, proceed to S13. In the case of No, the corresponding processing unit 53 provides the image as it is to the object recognition unit 54, and the series of processing ends.

In S13, the response processing unit 53 determines whether or not an abnormality has occurred in the frame one frame before the latest image acquired in S11. If the answer is Yes, proceed to S15. If the answer is No, proceed to S14.

In S14, the correspondence processing unit 53 replaces the high-brightness abnormal part in the latest image acquired in S11 with the image one frame before. The correspondence processing unit 53 provides the image after the replacement process to the object recognition unit 54. The series of processing ends at S14.

In S15, the correspondence processing unit 53 discards the latest image acquired in S11. The series of processing ends at S15.

According to the first embodiment described above, the high brightness abnormality caused by the incidence of the light pulse LP from the lidar 80 as a pulse emission type optical sensor is extracted using an extraction algorithm compatible with the shutter method of the in-vehicle camera 1. Ru. Therefore, high brightness abnormalities caused by the lidar 80 can be extracted according to the characteristics of the vehicle-mounted camera 1. Since the handling process for an image in which a high-brightness abnormality has occurred is different from that for other images, appropriate measures can be taken against the occurrence of a high-brightness abnormality caused by the lidar 80.

Further, according to the first embodiment, the extraction algorithm compatible with the rolling shutter method is based on detecting differences in characteristics between a plurality of frames using time series data of images including a plurality of consecutive frames. Contains an algorithm to extract abnormal images. Since the light pulse LP is incident instantaneously, by comparing with the previous and subsequent frames, etc., it is possible to improve the accuracy of extraction of the abnormality occurrence image.

Further, according to the first embodiment, the extraction algorithm compatible with the rolling shutter method detects high brightness abnormalities from a brightness value map in a direction orthogonal to a plurality of lines in which time differences occur in photographing timing in the image sensor of the in-vehicle camera 1. This includes an algorithm that extracts an abnormality image based on detecting the step shape of the brightness values shown. By accurately determining the characteristics of the high-luminance abnormality peculiar to the optical pulse LP and the rolling shutter, it is possible to improve the accuracy of extraction of an abnormality occurrence image.

Further, according to the first embodiment, the process for dealing with an abnormality image includes the process of deleting all frames corresponding to the abnormality image. By deleting the abnormality image, it is possible to appropriately restrict the use of the image for automatic driving.

Further, according to the first embodiment, the process for dealing with an abnormality image includes a process of transplanting a portion where a high-intensity abnormality has occurred from among the frames of the abnormality image from pixels corresponding to the previous frame. Through transplantation, an image in which an abnormality has occurred can be used as an image in which a high-intensity abnormality has been repaired.

Further, according to the first embodiment, the process for responding to an abnormality image includes a process of masking a part of the frame of the abnormality image in which a high-intensity abnormality has occurred so as to exclude it from the target to be used in automatic driving. include. By masking, it is possible to appropriately restrict the use of abnormal parts while leaving the possibility of using other parts of the image.

Further, according to the first embodiment, in the configuration in which reliability information is added to an image, the processing for responding to an abnormality image includes determining the reliability of a portion where a high-intensity abnormality has occurred in the frame of the abnormality image. Includes processing that lowers the reliability of other parts. In realizing autonomous driving, image information can be utilized to the fullest by referring to the reliability of each part of the image.

Further, according to the first embodiment, the process for responding to an abnormality image is performed by changing the exposure time from the next time onward to the exposure time that was adopted as the current photographing condition for the abnormality image for the in-vehicle camera 1. Includes processing to output a request to change. By changing the photographing conditions, it is possible to reduce the possibility that the high brightness abnormality will occur again after the next photographing.

Furthermore, the extraction algorithm corresponding to the rolling shutter method includes an algorithm that extracts images in which an abnormality has occurred based on detecting a change in the width of the portion in which a high brightness abnormality has occurred, which is a change according to the exposure time, by changing the exposure time in response to a request to change the exposure time. By deleting the images in which an abnormality has occurred, it is possible to appropriately restrict the use of the images in question for autonomous driving.

Further, according to the first embodiment, a driving action for automatic driving is derived based on the result of the response process. Since the process for dealing with an abnormality image is different from the process for other images, it is possible to appropriately deal with the abnormality image and suppress the occurrence of erroneous judgments during automatic driving.

(Second embodiment)
As shown in FIG. 10, the second embodiment is a modification of the first embodiment. The second embodiment will be described focusing on the differences from the first embodiment.

The vehicle-mounted camera 2 of the second embodiment (see also FIGS. 1 and 2) employs a global shutter method as a shutter method. In the global shutter method shown in FIG. 10, exposure of all pixels is started at the same time. That is, for this reason, the rolling shutter method is characterized in that all pixels are photographed at the same timing in the same one frame, and there is no time difference in photographing time.

The upper part of Figure 10 shows three captured images, specifically the N-1th frame, the Nth frame, and the N+1th frame. Here, N is a natural number equal to or greater than 2. The lower part of Figure 10 shows the exposure when the vehicle-mounted camera 2 periodically captures three frames. With a global shutter, exposure of all lines begins at the same time, and after the exposure time ETG, exposure of all lines ends at the same time. Capture is performed cyclically at a specified cycle.

Three vertical lines in the lower row indicate light pulses LP that are incident on the vehicle-mounted camera 2. The period of the light pulse LP incident on the vehicle-mounted camera 1 depends, for example, on the scanning period of the rider 80 of the other vehicle OV. The period of the light pulse LP that enters the vehicle-mounted camera 2 is different from the period of photographing. For example, the period of the light pulse LP is longer than the period of photographing. Then, the exposure time of only some of the frames coincides with the timing at which the light pulse LP enters the vehicle-mounted camera 2.

In the example of FIG. 10, the N-1th frame is affected by the light pulse LP, so that the entire image becomes a high-luminance region. On the other hand, since the N-th and N+1-th frames are not affected by the light pulse LP at all, the entire image becomes a low-luminance region.

The flare extraction unit 62 uses an extraction algorithm compatible with the global shutter method. The extraction algorithm adopts the first method described in the first embodiment, that is, the method of extracting an abnormality image based on time-series data of an image including a plurality of consecutive frames.

The response processing unit 63 executes one type of response processing for the abnormality image out of the first response process described in the first embodiment and the seventh response process described below.

The seventh response process is a process of correcting the brightness of the entire abnormality image so that it becomes darker. That is, the brightness value of the entire image is reduced. This correspondence processing can be applied when the upper part remains in the image, but cannot be applied when complete whiteout has occurred to the extent that no information remains. Even if the response processing unit 63 attempts to execute the seventh response process, if it is determined that the seventh response process is not applicable, the response processing unit 63 executes the first response process of deleting the frame. You may also do so.

According to the second embodiment described above, the extraction algorithm compatible with the global shutter method is based on detecting differences in characteristics between multiple frames using time-series data of images including multiple consecutive frames. , includes an algorithm for extracting anomaly images. Since the light pulse LP is incident instantaneously, by comparing with the previous and subsequent frames, etc., it is possible to improve the accuracy of extraction of the abnormality occurrence image.

Furthermore, according to the second embodiment, the processing for dealing with an abnormality image includes a process of correcting so as to reduce the brightness value of the entire abnormality image. By lowering the overall brightness value, if information remains in the image, it becomes possible to use it for object recognition in the same way as a normal image.

(Third embodiment)
As shown in FIGS. 11 to 13, the third embodiment is a modification of the first embodiment. The third embodiment will be described focusing on the differences from the first embodiment.

The processing device 210 of the third embodiment handles flare (high brightness abnormality) that occurs when the light pulse LP emitted from the lidar 3 of the own vehicle EV is reflected by an object and enters the on-vehicle camera 1 of the own vehicle EV. Compatible processing has been realized.

A scene in which rider-induced flare of the own vehicle EV is likely to occur is, for example, as shown in FIG. 11, a scene where the own vehicle EV and another vehicle OV run parallel to each other on a road with multiple lanes on each side. For example, when the lidar 3 and the vehicle-mounted camera 1 are mounted toward the side of the vehicle EV, the light pulse LP emitted from the lidar 3 is reflected by another vehicle OV running parallel to the own vehicle EV, and the light pulse LP is reflected by the vehicle-mounted camera 1. It may be incident.

The vehicle-mounted camera 1 of the third embodiment uses a rolling shutter system as the shutter system.

As shown in FIG. 12, the processing device 210 includes an image acquisition section 51, a rider motion acquisition section 256, a flare extraction section 252, a correspondence processing section 253, an object recognition section 54, and a driving planning section 55, and a processor 10b that executes a program. This is a configuration that is included as a functional block realized by. The image acquisition unit 51, object recognition unit 54, and driving planning unit 55 are the same as those in the first embodiment.

The rider motion acquisition unit 256 acquires the motion of the rider 3 of the own vehicle EV. The operation of the lidar 3 includes information on the operation of the lidar 3, such as the scanning direction of the optical pulse LP and the timing of light emission.

The flare extraction unit 252 uses an extraction algorithm that supports the rolling shutter method. The extraction algorithm uses the following fourth method as the extraction method.

The fourth method is a method of determining whether or not the light pulse LP is incident on the vehicle-mounted camera 1 within the exposure time of the vehicle-mounted camera 1, using information on the movement of the lidar 3 and the like. Specifically, the flare extraction unit 252 estimates the generation position and generation timing of the reflected light reflected from the object where the light pulse LP is recognized, from the scanning direction and emission timing of the light pulse LP, and the immediately preceding object recognition information. do. Then, the flare extraction unit 252 determines whether the position where the reflected light is generated is within the field of view of the vehicle-mounted camera 1 within the exposure time of the rolling shutter.

Here, the speed of light is sufficiently faster than the control speed of the vehicle-mounted camera 1. Therefore, the optical path along which the reflected light travels and the time it takes to travel along the optical path can be ignored in the estimation. Further, the object recognition information may be acquired from the object recognition section 54 or may be extracted from the image of the vehicle-mounted camera 1.

Here, the flare extracting unit 52 may specify a line from which reflected light is estimated to be photographed among a plurality of lines of pixels in the horizontal direction. If the flare extraction unit 52 determines that the position where the reflected light is generated is within the field of view of the in-vehicle camera 1 within the exposure time, or that there is a line where the reflected light is photographed, the flare extraction unit 52 configures the frame. The image that appears is determined to be an image in which an abnormality has occurred.

The extraction algorithm may be configured to improve accuracy by combining the fourth method and any one of the first to third methods.

The response processing unit 253 performs one of the first to sixth response processes described in the first embodiment and the eighth and ninth response processes described below as a response process for an image in which an abnormality has occurred.

The eighth corresponding process is a process of shifting the light emission timing of the lidar 3 so as to suppress the interference of the light pulse LP of the lidar 3 after the next photographing. That is, the correspondence processing unit 253 acquires information on the shooting timing for the next frame and subsequent frames scheduled for the vehicle-mounted camera 1 and information on the light emission timing scheduled for the rider 3.

Then, the corresponding processing unit 253 shifts the light emission timing of the rider 3 so that the light emission timing of the rider 3 occurs outside the exposure time of the vehicle-mounted camera 1. The response processing unit 253 may transmit a light emission timing change request to the rider 3. If the processing device 210 is the main entity that plans and controls the movement of the lidar 3, the corresponding processing unit 253 may reflect the change in the light emission timing in the lidar movement plan by the processing device 210.

The ninth response process is a process for controlling the light emission power of the LIDAR 3 so that it is less than the threshold at which a high brightness abnormality occurs in the vehicle-mounted camera 1. By making the LIDAR 3 emit weak light, the image can be kept normal even if a light pulse LP is incident on the vehicle-mounted camera 1 in shooting the next frame or later.

Furthermore, the response process may be a combination of one of the eighth and ninth response processes and one of the first to fifth response processes described in the first embodiment.

Next, an example of a processing method of extracting an abnormality image and performing corresponding processing according to the extraction result among the processing performed by the processing device 210 will be described using the flowchart of FIG. 13. The series of processes shown in steps S11 to S15 are executed, for example, each time the in-vehicle camera 1 generates one frame of image, or each time the processing device 210 starts updating the automatic driving plan.

In S21, the image acquisition unit 51 acquires the latest image (for example, one frame) from the vehicle-mounted camera 1. After processing in S21, the process advances to S22.

In S22, the flare extraction unit 252 determines whether it is estimated that reflected light from the rider 3 of the own vehicle EV is captured within the angle of view of the image acquired in S21. If Yes, the flare extraction unit 252 proceeds to S23 where flare is occurring. In the case of No, the corresponding processing unit 253 provides the image as is to the object recognition unit 54, and the series of processing ends.

In S23, the correspondence processing unit 253 determines whether the frame one frame before the latest image acquired in S21 is normal. If Yes, proceed to S24. If No, proceed to S25.

In S24, the correspondence processing unit 253 replaces the high-brightness abnormal part in the latest image acquired in S21 with the image one frame before. The correspondence processing unit 253 provides the image after the replacement process to the object recognition unit 54. After processing in S24, the process advances to S26.

In S25, the correspondence processing unit 253 discards the latest image acquired in S21. After processing in S25, the process advances to S26.

In S26, the response processing unit 253 changes the operation of the rider 3. Specifically, the above-mentioned change to shift the light emission timing or change to cause the rider 3 to emit weak light is implemented. The series of processing ends at S26.

According to the third embodiment described above, the extraction algorithm compatible with the rolling shutter method uses the information on the operation of the lidar 3 as a pulse emission type optical sensor to detect the light pulse LP mounted on the vehicle within the exposure time of the vehicle-mounted camera 1. It includes an algorithm for extracting images in which an abnormality has occurred based on determining whether the light is incident on the camera 1 or not. By such a determination, a high brightness abnormality caused by the lidar 3 can be determined with extremely high accuracy.

Further, according to the third embodiment, the process for responding to an abnormality image includes a process for requesting to shift the light emission timing of the lidar 3 so as to suppress interference with the light pulse LP of the lidar 3 from the next shooting onward. include. It is possible to suppress the occurrence of high brightness abnormalities after the next photographing due to a shift in the light emission timing.

Further, according to the third embodiment, the processing for responding to the abnormality occurrence image includes processing for controlling the light emitting power of the lidar 3 so that it is smaller than the threshold value at which a high-intensity abnormality occurs in the vehicle-mounted camera 1. By causing the lidar 3 to emit weak light, even if the light pulse LP were to be incident on the vehicle-mounted camera 1, the possibility that a high brightness abnormality would occur can be reduced.

(Fourth embodiment)
The fourth embodiment is a modification of the third embodiment. The fourth embodiment will be described focusing on the differences from the third embodiment.

The processing device 210 of the fourth embodiment, like the third embodiment, realizes processing to deal with flare (high brightness abnormality) that occurs when a light pulse LP emitted from the lidar 3 of the host vehicle EV is reflected by an object and enters the vehicle-mounted camera 2 of the host vehicle EV. However, the vehicle-mounted camera 2 of the fourth embodiment (see also FIG. 12) employs the same global shutter method as the second embodiment as the shutter method.

The flare extraction unit 262 uses an extraction algorithm compatible with the global shutter method. The extraction algorithm employs the following fifth method as an extraction method.

The fifth method is a method of determining whether or not the light pulse LP is incident on the vehicle-mounted camera 2 within the exposure time of the vehicle-mounted camera 2 using information on the operation of the lidar 3, etc. However, since the fifth method is optimized for the global shutter method, it does not go as far as identifying the line on which the reflected light is captured, as in the fourth method.

The extraction algorithm may be configured to improve accuracy by combining the fifth method and the first method.

The response processing unit 263 performs the first response process described in the first embodiment, the seventh response process described in the second embodiment, and the eighth response process described in the third embodiment as the response process for the abnormality image. Execute one type of corresponding processing from ~9. Further, the corresponding process may be a combination of one type of the eighth and ninth corresponding processes and one type of the first and seventh corresponding processes.

(Other embodiments)
Although multiple embodiments have been described above, the present disclosure is not to be construed as being limited to those embodiments, and may be applied to various embodiments and combinations within the scope of the gist of the present disclosure. Can be done.

The extraction algorithm may be an algorithm using a neural network. For example, when an image is input to a neural network, a result indicating whether the image is an abnormal image may be output.

As another embodiment, in the third embodiment, the flare extraction unit 252 may employ the same extraction algorithm as in the first embodiment, that is, an algorithm that does not use information about the rider 3.

As another embodiment, in the eighth response process, the response processing unit 253 may shift the photographing timing of the vehicle-mounted camera 1 instead of the light emission timing of the rider 3. The corresponding processing unit 253 may shift both the light emission timing of the rider 3 and the photographing timing of the vehicle-mounted camera 1.

As another embodiment, the rider motion acquisition unit 256 in the processing device 210 acquires the motion of the rider 80 of the other vehicle OV using V2V communication, and uses this to cause the rider to take the same response as the rider 3 of the own vehicle EV. 80 may be implemented.

As another embodiment, as shown in FIG. 14, the automatic driving system may include a plurality of ECUs 310 and 312. For example, the camera ECU 310 is an ECU that controls the vehicle cameras 1 and 2 and performs image processing. AD-ECU 312 is an ECU that executes automatic driving.

Camera ECU 310 is realized, for example, mainly by a computer. A computer that implements camera ECU 310 may include at least one memory 310a and at least one processor 310b. The memory 310a is at least one type of non-transitional physical storage medium, such as a semiconductor memory, a magnetic medium, an optical medium, etc., that non-temporarily stores programs, data, etc. that can be read by the processor 310b. good. Furthermore, a rewritable volatile storage medium such as a RAM (Random Access Memory) may be provided as the memory 310a. The processor 310b includes, as a core, at least one type of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU.

The camera ECU 310 may be configured to include an image acquisition section 51, flare extraction sections 52, 62, 252, 262, and corresponding processing sections 53, 63, 253, 263 among the functional blocks of the processing device 10, 210.

The AD-ECU 312 is realized mainly by a computer, for example. A computer implementing the AD-ECU 312 may include at least one memory 312a and at least one processor 312b. The memory 312a is at least one type of non-transitional physical storage medium, such as a semiconductor memory, a magnetic medium, an optical medium, etc., that non-temporarily stores programs, data, etc. that can be read by the processor 312b. good. Furthermore, a rewritable volatile storage medium such as a RAM (Random Access Memory) may be provided as the memory 312a. The processor 312b includes, as a core, at least one type of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU.

The AD-ECU 312 may include the object recognition unit 54 and the driving planning unit 55 among the functional blocks of the processing device 10, 210. Note that the object recognition unit 54 may be included on the camera ECU 310 side.

As another embodiment, the vehicle-mounted cameras 1 and 2 may further include a bandpass filter, for example in the optical system. The bandpass filter has a function of attenuating light with a wavelength of 800 to 1000 nm, which is often used in the optical pulse LP emitted by the lidar 3, 80. The bandpass filter may have a spectral characteristic such that the energy transmittance of light with a wavelength of 800 to 1000 nm is smaller than 0.01.

As another embodiment, the in-vehicle camera 401 shown in FIG. 15 may further include a polarizing filter 402 in the optical system, for example. The polarizing filter 402 is provided, for example, at the tip of the lens barrel 403 of the vehicle-mounted camera 401. As shown in FIG. 16, the polarizing filter 402 has an absorption axis 402a and a transmission axis 402b that are orthogonal to each other, transmits light Pb polarized in the direction of the transmission axis 402b, and absorbs light Pa polarized in the direction of the absorption axis 401a. have the property of

The absorption axis 402a of the polarizing filter 402 is set along the expected polarization direction of the reflected light in order to efficiently absorb the reflected light caused by the light pulse LP of the lidar 3 being reflected by an object. That is, on the premise of this example, the light source 3a of the rider 3 of the host vehicle EV emits a linearly polarized light pulse LP. For example, when the lidar 3 emits a light pulse LP that is linearly polarized in the polarization direction along the horizontal direction, the absorption axis 402a of the polarizing filter 402 is arranged along the horizontal direction, and the transmission axis 402b is arranged along the vertical direction. It is arranged like this. By doing so, only the polarized light Pb along the vertical direction is incident on the image sensor 404. That is, it becomes difficult for the light pulse LP of the lidar 3 to reach the image sensor 404.

In another embodiment, the processing device 10, 210 may be applied to a vehicle capable of performing autonomous driving level 1 or 2 driving assistance.

The control unit and techniques described in this disclosure may be implemented by a special purpose computer comprising a processor programmed to perform one or more functions embodied by a computer program. Alternatively, the apparatus and techniques described in this disclosure may be implemented with dedicated hardware logic circuits. Alternatively, the apparatus and techniques described in this disclosure may be implemented by one or more special purpose computers configured by a combination of a processor executing a computer program and one or more hardware logic circuits. The computer program may also be stored as instructions executed by a computer on a computer-readable non-transitory tangible storage medium.

(Disclosure of technical ideas)
This specification discloses multiple technical ideas described in multiple sections listed below. Some sections may be written in a multiple dependent form, in which subsequent sections alternatively cite preceding sections. The terms written in these multiple dependent forms define multiple technical ideas.

<Technical philosophy 1>
In a vehicle (EV) capable of autonomous driving, at least one processor (10b , 310b),
The at least one processor includes:
obtaining the image;
From the image, an extraction algorithm corresponding to the shutter method is used to determine the occurrence of an abnormality in which a high-intensity abnormality occurs due to the light pulse (LP) from the pulse-emitting optical sensor (3, 80) entering the vehicle-mounted camera. extracting the image;
A processing device configured to perform the following steps: making the handling process for the abnormality image different from the handling process for other images.

<Technical philosophy 2>
The shutter method is a rolling shutter method,
The extraction algorithm includes an algorithm for extracting the abnormality image based on detecting a difference in characteristics between the plurality of frames using time-series data of the image including a plurality of consecutive frames. The processing device according to Concept 1.

<Technical philosophy 3>
The shutter method is a rolling shutter method,
The extraction algorithm is based on detecting a stepped shape of brightness values indicating a high brightness abnormality from a brightness value map in a direction perpendicular to a plurality of lines in which time differences occur in photographing timing in the image sensor of the in-vehicle camera. The processing device according to technical idea 1 or 2, which includes an algorithm for extracting an abnormality image.

<Technical philosophy 4>
The shutter method is a rolling shutter method,
The processing device according to any one of technical ideas 1 to 3, wherein the processing for responding to the abnormality image includes a process of deleting all frames corresponding to the abnormality image.

<Technical philosophy 5>
The shutter method is a rolling shutter method,
Technical Ideas 1 to 3, wherein the handling process for the abnormality image includes a process of transplanting a part where a high-intensity abnormality has occurred from among the frames of the abnormality image from pixels corresponding to the previous frame. The processing device according to any one of the above.

<Technical philosophy 6>
The shutter method is a rolling shutter method,
The technical idea is that the processing for responding to the abnormality image includes a process of masking a part of the frame of the abnormality image in which a high-intensity abnormality has occurred so as to exclude it from the target to be used in the automatic driving. The processing device according to any one of items 1 to 3 and 5.

<Technical philosophy 7>
The at least one processor includes:
The image processing apparatus is configured to further perform the step of adding reliability information of the image to the image;
The shutter method is a rolling shutter method,
Technical concept 1, wherein the processing for responding to the abnormality image includes a process of lowering the reliability of a portion where a high-intensity abnormality has occurred in the frame of the abnormality image than the reliability of other portions. The processing device according to any one of items 3, 5, and 6.

<Technical philosophy 8>
The shutter method is a rolling shutter method,
The processing for responding to the abnormality image includes requesting the in-vehicle camera to change the exposure time for the next time the image is taken, with respect to the exposure time currently adopted as the photographing condition for the abnormality image. The processing device according to any one of Technical Ideas 1 to 3 and 5 to 7, which includes output processing.

<Technical philosophy 9>
The extraction algorithm is an algorithm that extracts the image in which the abnormality occurs based on detecting a change in the width of the portion where the high-intensity abnormality occurs according to the exposure time by changing the exposure time in response to the request. The processing device according to Technical Idea 8, comprising:

<Technical Thought 10>
The at least one processor includes:
further configured to obtain an operation of the pulsed light emitting optical sensor;
The shutter method is a rolling shutter method,
The extraction algorithm is based on determining whether or not the light pulse is incident on the vehicle-mounted camera within the exposure time of the vehicle-mounted camera, using information on the operation of the pulse-emitting optical sensor. The processing device according to any one of Technical Ideas 1 to 9, which includes an algorithm for extracting a generated image.

<Technical Thought 11>
The shutter method is a global shutter method,
The extraction algorithm includes an algorithm for extracting the abnormality image based on detecting a difference in characteristics between the plurality of frames using time-series data of the image including a plurality of consecutive frames. The processing device according to Concept 1.

<Technical Thought 12>
The shutter method is a global shutter method,
The processing device according to Technical Idea 1 or 11, wherein the process for responding to the abnormality image includes a process of deleting all frames corresponding to the abnormality image.

<Technical Thought 13>
The shutter method is a global shutter method,
The processing device according to Technical Idea 1 or 11, wherein the processing for responding to the abnormality image includes a process of correcting so as to reduce a luminance value of the entire abnormality image.

<Technical Thought 14>
The at least one processor includes:
further configured to obtain the operation of the pulsed light emitting optical sensor;
The shutter method is a global shutter method,
The extraction algorithm is based on determining whether or not the light pulse is incident on the vehicle-mounted camera within the exposure time of the vehicle-mounted camera, using information on the operation of the pulse-emitting optical sensor. The processing device according to any one of Technical Ideas 1 and 11 to 13, which includes an algorithm for extracting a generated image.

<Technical Thought 15>
The response process to the abnormality image is a process of requesting that the light emission timing of the pulse emission type optical sensor be shifted so as to suppress interference with the light pulse of the pulse emission type optical sensor from the next shooting onward. The processing device according to Technical Idea 10 or 14, comprising:

<Technical Thought 16>
Technical idea 10 or 14, wherein the processing for responding to the abnormality occurrence image includes processing for controlling the light emission power of the pulsed light emission type optical sensor so that the processing corresponds to the abnormality occurrence image is smaller than a threshold value at which a high-intensity abnormality occurs in the vehicle-mounted camera. The processing device described in .

<Technical Thought 17>
The at least one processor includes:
The processing device according to any one of technical ideas 1 to 16, further configured to perform the following: deriving the driving action of the automatic driving based on the result of the corresponding processing.

<Technical Thought 18>
In a vehicle (EV) capable of autonomous driving, at least one processor (10b , 310b), the processing method being performed by
obtaining the image;
From the image, an extraction algorithm corresponding to the shutter method is used to determine the occurrence of an abnormality in which a high-intensity abnormality occurs due to the light pulse (LP) from the pulse-emitting optical sensor (3, 80) entering the vehicle-mounted camera. extracting the image;
A processing method comprising: making the handling process for the abnormality image different from the handling process for other images.

<Technical Thought 19>
An automatic driving system,
An on-vehicle camera (1, 2, 401) that is installed in a vehicle (EV) capable of autonomous driving, has a predetermined shutter method, and captures images;
A processing device (10, 210, 310) including at least one processor (10b, 310b) in order to reflect the corresponding processing for the image taken by the on-vehicle camera in the driving action of the vehicle,
The at least one processor includes:
obtaining the image;
From the image, an extraction algorithm corresponding to the shutter method is used to determine the occurrence of an abnormality in which a high-intensity abnormality occurs due to the light pulse (LP) from the pulse-emitting optical sensor (3, 80) entering the vehicle-mounted camera. extracting the image;
An automatic driving system configured to perform the following steps: making the corresponding processing for the abnormality occurrence image different from the corresponding processing for other images.

<Technical Concept 20>
The pulse-emitting optical sensor emits the light pulse linearly polarized in a predetermined polarization direction,
The autonomous driving system described in Technical Idea 19, wherein the onboard camera is equipped with a polarizing filter (402) configured to absorb at least a portion of the light pulses linearly polarized in the polarization direction.

1, 2, 401: Vehicle-mounted camera, 3, 80: Lidar (pulse emission type optical sensor), 10, 210: Processing device, 310: Camera ECU (processing device), 10b, 310b: Processor, EV: Vehicle, LP: light pulse

Claims (17)

In a vehicle (EV) capable of autonomous driving, at least one processor (10b , 310b),
The at least one processor includes:
obtaining the image;
From the image, an extraction algorithm corresponding to the shutter method is used to determine the occurrence of an abnormality in which a high-intensity abnormality occurs due to the light pulse (LP) from the pulse-emitting optical sensor (3, 80) entering the vehicle-mounted camera. extracting the image;
A processing device configured to perform the following steps: making the handling process for the abnormality image different from the handling process for other images. The shutter method is a rolling shutter method,
The extraction algorithm includes an algorithm that extracts the abnormality image based on detecting a difference in characteristics between the plurality of frames using time-series data of the image including a plurality of consecutive frames. Item 1. The processing device according to item 1. The shutter method is a rolling shutter method,
The extraction algorithm is based on detecting a stepped shape of brightness values indicating a high brightness abnormality from a brightness value map in a direction perpendicular to a plurality of lines in which time differences occur in photographing timing in the image sensor of the in-vehicle camera. The processing device according to claim 1, comprising an algorithm for extracting an abnormality occurrence image. The shutter method is a rolling shutter method,
2. The processing device according to claim 1, wherein the processing for responding to the abnormality image includes a process of deleting all frames corresponding to the abnormality image. The shutter method is a rolling shutter method,
2. The process of responding to the abnormality image includes a process of transplanting a portion where a high-intensity abnormality has occurred from a pixel corresponding to a previous frame in a frame of the abnormality image. Processing equipment. The shutter method is a rolling shutter method,
1 . The process of responding to the abnormality image includes a process of masking a part of the frame of the abnormality image in which a high-intensity abnormality occurs so as to exclude it from a target to be used in the automatic driving. The processing device described in . The at least one processor includes:
The image processing apparatus is configured to further perform the step of adding reliability information of the image to the image;
The shutter method is a rolling shutter method,
2. The method according to claim 1, wherein the processing for responding to the abnormality image includes a process of lowering reliability of a portion where a high-intensity abnormality has occurred in a frame of the abnormality image than the reliability of other portions. Processing equipment as described. The shutter method is a rolling shutter method,
The processing for responding to the abnormality image includes requesting the in-vehicle camera to change the exposure time for the next time the image is taken, with respect to the exposure time currently adopted as the photographing condition for the abnormality image. The processing device according to claim 1, comprising processing for outputting. The extraction algorithm is an algorithm that extracts the abnormality occurrence image based on detecting a change in the width of the portion where the high brightness abnormality occurs according to the exposure time by changing the exposure time corresponding to the request. The processing device according to claim 8, comprising: The at least one processor includes:
further configured to obtain the operation of the pulsed light emitting optical sensor;
The shutter method is a rolling shutter method,
The extraction algorithm is based on determining whether or not the light pulse is incident on the vehicle-mounted camera within the exposure time of the vehicle-mounted camera, using information on the operation of the pulse-emitting optical sensor. Processing device according to claim 1, comprising an algorithm for extracting the generated image. The shutter method is a global shutter method,
The extraction algorithm includes an algorithm that extracts the abnormality image based on detecting a difference in characteristics between the plurality of frames using time-series data of the image including a plurality of consecutive frames. Item 1. The processing device according to item 1. The shutter method is a global shutter method,
2. The processing device according to claim 1, wherein the processing for responding to the abnormality image includes a process of deleting all frames corresponding to the abnormality image. The shutter method is a global shutter method,
2. The processing device according to claim 1, wherein the processing for responding to the abnormality image includes a process of correcting so as to reduce a luminance value of the entire abnormality image. The at least one processor includes:
further configured to obtain the operation of the pulsed light emitting optical sensor;
The shutter method is a global shutter method,
The extraction algorithm is based on determining whether or not the light pulse is incident on the vehicle-mounted camera within the exposure time of the vehicle-mounted camera, using information on the operation of the pulse-emitting optical sensor. 2. Processing device according to claim 1, comprising an algorithm for extracting generated images. The response process to the abnormality image is a process of requesting the light emission timing of the pulse light emitting optical sensor to be shifted so as to suppress interference with the light pulse of the pulse light emitting optical sensor from the next shooting onward. The processing device according to claim 10 or 14, comprising: 15. The process according to claim 10 or 14, wherein the process for responding to the abnormality image includes a process for controlling the light emission power of the pulsed light emitting optical sensor so that it is smaller than a threshold value at which a high brightness abnormality occurs in the in-vehicle camera. Processing equipment as described. The at least one processor includes:
The processing device according to claim 1, further configured to derive the driving action for the automatic driving based on the result of the corresponding processing.

Images