JP2018074757A - Patrol inspection system, information processing apparatus, and patrol inspection control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect an abnormal location that occurs in a power transmission facility, stably through image processing.SOLUTION: A patrol inspection system performs a patrol inspection upon an inspected object based on image data that are picked up by a camera mounted on an aircraft. The patrol inspection system includes a management device and an information processing apparatus. The management device includes learning model generation means which generates a learning model for detecting abnormality that occurs in the inspected object, from among images. The information processing apparatus includes storage means, input means, abnormal location detection means and abnormal location display means. The storage means stores a learning model that is generated by the management device. The input means inputs the image data. The abnormal location detection means detects image data in an abnormal location where abnormality occurs in the inspected object, from the image data based on the learning model. The abnormal location display means displays an image in the abnormal location based on the image data in the abnormal location.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a patrol inspection system, an information processing apparatus, and a patrol inspection control program using a flying object.

  Conventionally, regular inspections of power transmission facilities such as overhead power transmission lines (hereinafter referred to as power transmission lines) and patrols at the time of accidents are carried out by a patrolman riding a helicopter and flying along the power transmission lines and looking at the power transmission lines with binoculars. While checking the presence or absence of abnormal parts, or while moving the power transmission equipment on foot.

  In addition, an image of a power transmission line is taken from a helicopter, and a patrolman looks at the photographed image to check whether the power transmission line is broken or an arc mark (lightning mark) is present.

JP 2005-57956 A

  However, inspection / patrolling with binoculars by a patrolman is a burden on the patrolist and there is a risk of overlooking an abnormal part.

  In addition, inspecting and patrol of power transmission equipment using a helicopter, in addition to the cost of using the helicopter, a plurality of patrolmen are boarded in addition to the helicopter pilot, so the cost of inspection / patrol is high. In addition, it is not possible to easily and quickly respond to sudden inspection / patrol.

  In addition, the inspection work of the power transmission line by image photographing also has a problem that a quick and accurate inspection cannot be performed because the patrol person detects an abnormal part while viewing all the photographed images. Furthermore, a technique for detecting wire breakage and arc traces using image processing technology for images is also considered, but the quality of the captured image is not constant due to changes in the shooting environment, and detection is also possible. There are various states of abnormal places such as wire breaks and arc marks, and it is difficult to stably detect abnormal places.

  The problem to be solved by the present invention is to provide a patrol inspection system, an information processing apparatus, and a patrol inspection control program capable of stably detecting an abnormal portion generated in a power transmission facility by image processing.

  According to the embodiment, the inspection inspection system performs inspection inspection on an inspection object based on image data captured by a camera mounted on an aircraft or the like. The inspection inspection system includes a management device and an information processing device. The management device includes a learning model generation unit that generates a learning model for detecting an abnormality occurring in the inspection object from an image. The information processing apparatus includes a storage unit, an input unit, an abnormal part detection unit, and an abnormal part display unit. The storage means stores the learning model generated by the management device. The input means inputs the image data. The abnormal part detection means detects image data of an abnormal part where an abnormality has occurred in the inspection object from the image data based on the learning model. The abnormal part display means displays an image of the abnormal part based on the image data of the abnormal part.

The block diagram which shows the structure of the patrol inspection system in this embodiment. The block diagram which shows the structure of the patrol inspection system in this embodiment. The block diagram which shows the structure of the information processing apparatus in this embodiment. The block diagram which shows the function structure in the information processing apparatus in this embodiment. The figure which shows an example of the external appearance of the drone in this embodiment. The block diagram which shows the main structures of the drone in this embodiment. The block diagram which shows the function structure in the management apparatus in this embodiment. The flowchart for demonstrating the learning model production | generation process performed by the management apparatus in this embodiment. The figure which shows an example of the area | region of the abnormal image in this embodiment, and the area | region of a normal image. The figure which shows an example of the area | region of the abnormal image in this embodiment, and the area | region of a normal image. The figure which shows an example of the area | region of the abnormal image in this embodiment, and the area | region of a normal image. The figure which shows notionally the combination of the abnormal image and normal image by the normal image / abnormal image combination control part in this embodiment. The figure which shows the relationship of the production | generation image conditions memorize | stored by the parameter | index / condition relevant memory | storage part in this embodiment, and a recall. The figure which shows the relationship between the production | generation image conditions memorize | stored by the parameter | index / condition relevant memory | storage part in this embodiment, and a relevance rate. The figure which shows an example of the patrol / inspection route selection screen in this embodiment. The figure for demonstrating the flight plan of the drone in this embodiment. The figure for demonstrating the flight plan of the drone in this embodiment. The figure which shows an example of the to-be-inspected object provided in the power transmission installation in this embodiment. The figure which shows an example of the to-be-inspected object provided in the power transmission installation in this embodiment. The figure which shows an example of the to-be-inspected object provided in the power transmission installation in this embodiment. The figure which shows an example of the flight plan (flight control data) produced | generated by the flight plan process part in this embodiment. The figure which shows an example of the acquisition data produced | generated by the acquisition data production | generation part in this embodiment. The figure which shows an example of the acquisition data produced | generated by the acquisition data production | generation part in this embodiment. The figure which shows an example of the inspection object item selection screen in this embodiment. 6 is a flowchart showing acquired data management processing in the information processing apparatus according to the embodiment. The flowchart explaining the abnormal location detection process in this embodiment. The figure which shows an example of the abnormality candidate image display screen in this embodiment. The figure which shows an example of the abnormal location confirmation screen in this embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
FIG.1 and FIG.2 is a block diagram which shows the structure of the patrol inspection system 1 in this embodiment. A patrol inspection system 1 according to the present embodiment includes, for example, an aircraft such as an unmanned aerial vehicle (hereinafter referred to as a drone 14) at a normal time or when an accident occurs with respect to a power transmission facility (transmission line, steel tower) 17 for transmitting power from a power plant or substation. It is a system for patrol inspection using. The inspection inspection system 1 is based on the drone 14 and a plurality of different inspection objects (for example, overhead ground wire, power transmission line, etc.) included in the power transmission facility 17 based on the image data captured by the camera mounted on the drone 14. The inspection inspection control system 2 is configured to detect an abnormal point generated in an insulator, a flashing indicator, an arc horn, a steel tower, a steel tower foundation concrete, and the like, and record the detection result as a inspection inspection result report. In the present embodiment, the inspection inspection control system 2 is configured by a management device 12 connected to the information processing device 10 via the network 15, for example. However, the inspection inspection control system 2 can be realized as one apparatus. Further, the management device 12 is realized by cloud computing, and may be realized by one server or a plurality of servers connected via the network 15 (Internet) operating in cooperation. good.

As shown in FIGS. 1 and 2, the inspection inspection system 1 includes an inspection inspection control system 2 (information processing apparatus 10 and management apparatus 12) and a drone 14.
The information processing apparatus 10 manages / controls navigation of the drone 14, organizes / manages acquisition data (including image data (moving images, still images), position data, etc.) acquired from the drone 14, and acquires from the drone 14. Processes such as inspection / inspection management that summarize the processing results for data are executed. For example, the information processing apparatus 10 is used by being transported to the vicinity of the power transmission facility 17 to be inspected together with the drone 14, and detecting an abnormal part generated in the inspection object based on the image data photographed by the drone 14. Then, an abnormal part image confirmation process is executed.

  Based on the image data in the information processing apparatus 10, the management apparatus 12 generates a learning model to be used for image processing for detecting an abnormal part generated in the inspection object, and the processing result created in the information processing apparatus 10. Perform recording etc.

  The drone 14 generates position data generated using a Global Navigation Satellite System (GNSS) based on a flight plan (flight control data) for patrol the power transmission facility 17 set by the information processing apparatus 10. For example, autonomous flight is performed using position data generated from a GPS signal received from a GPS (Global Positioning System) satellite 16, and the power transmission equipment 17 to be inspected for inspection is photographed and image data is stored. The drone 14 can receive flight control data from the information processing apparatus 10 in real time during the flight, and can fly by manual control in accordance with an operation performed by the operator on the information processing apparatus 10.

  FIG. 3 is a block diagram illustrating a configuration of the information processing apparatus 10 according to the present embodiment. The information processing apparatus 10 is realized by a personal computer, for example. The information processing apparatus 10 includes a processor 10a, a memory 10b, a storage device 10c, an input / output interface (I / F) 10d, a display device 10e, an input device 10f, a wireless communication device 10g, and a communication device 10h.

  The processor 10a executes a basic program (OS) and application programs stored in the memory 10b to realize various functions. For example, the processor 10a manages the flight of the drone 14 by executing a patrol inspection control program, and detects an abnormal point generated in the inspection object of the power transmission facility based on the image taken by the drone 14. Based on the detection result, a process for collecting the inspection inspection results is executed (see FIG. 4).

  The memory 10b stores programs executed by the processor 10a, temporary data, and the like.

  The storage device 10c stores various programs and various data. The data stored in the storage device 10c includes flight control data for controlling the navigation of the drone 14, acquired data (image data, position data) received from the drone 14, and abnormalities occurring in the inspection object from the image data. It includes learning model data used for image processing for detecting a location, image data of an abnormal location, inspection inspection data that summarizes inspection inspection results, and the like.

  The input / output I / F 10d is an interface for transmitting / receiving data to / from an external device. The input / output I / F 10d can input / output data via a portable memory medium, for example.

  The display device 10e is an LCD (Liquid Crystal Display) or the like, and displays a screen corresponding to the processing of the processor 10a. The input device 10f is a keyboard or a pointing device, and is operated by an operator or the like.

  The wireless communication device 10g controls wireless communication, and controls wireless communication with the drone 14 or with a base station accommodated in a wireless public network.

  The communication device 10 h controls communication with external devices such as the management device 12, the drone 14, and other information processing devices 10 through the network 15. The network 15 includes a wireless or wired WAN (Wide Area Network), a LAN (Local Area Network), and the like.

  FIG. 4 is a block diagram showing a functional configuration realized by executing the inspection inspection control program by the processor 10a in the information processing apparatus 10 according to the present embodiment.

  The information processing apparatus 10 realizes the functions of, for example, the navigation management unit 20, the acquired data arrangement management unit 21, and the inspection inspection management unit 22 based on the inspection inspection control program.

  The navigation management unit 20 manages / controls the navigation of the drone 14, and includes a patrol inspection instruction input unit 20a and a flight plan processing unit 20b.

The inspection inspection instruction input unit 20a inputs an instruction about the contents of inspection inspection according to the selection on the selection menu displayed on the display device 10e, for example, by an input operation on the input device 10f by an operator. The contents of the patrol inspection include, for example, designation of the power transmission equipment 17 that is a target of the patrol inspection, designation of an inspection object included in the power transmission equipment 17 that is a target of inspection, and an abnormal type (such as an arc mark) that is a target of inspection.

  The flight plan processing unit 20b causes the drone 14 to navigate a flight plan according to the content of the inspection inspection input in the inspection inspection instruction input unit 20a, that is, a route that can photograph the inspection object to be inspected. Generate flight control data. The inspection inspection instruction input unit 20a can generate a flight plan using information acquired from the outside through the network 15, such as weather information and map information.

  The acquired data organization management unit 21 organizes / manages acquired data (including image data (moving images, still images), position data, etc.) acquired from the drone 14, and includes a learning model storage unit 21a, a detection target Instruction input unit 21b, abnormal part detection part 21c, acquired data storage part 21d, image confirmation part 21e, abnormal part display part 21f, abnormal part storage part 21g, confirmation instruction input part 21h, confirmation data storage part 21k, and abnormal part image A data storage unit 21m is included.

  The learning model storage unit 21 a stores the learning model generated by the management device 12. The learning model is for detecting an abnormality occurring in the inspection object from the image data acquired from the drone 14 from the image, and represents a characteristic of an abnormal state occurring in the inspection object. In the management device 12, an abnormality occurred in the inspection object so that the image data is not constant due to a change in the photographing environment photographed by the drone 14, or the abnormal part to be detected is in various states. A learning model capable of reliably detecting the location is generated (details of generation of the learning model will be described later). As an abnormality that has occurred in the inspection object, for example, in the case of a power transmission line, there is an arc mark. Since the arc mark is an abnormality that occurs when a lightning strike occurs on the power transmission line, it is not an event that frequently occurs, but there are few samples of image data obtained by photographing the actual state. Therefore, it is difficult to reliably detect the abnormal part from the image data photographed by the drone 14 only with the learning model based on the image data of the abnormal part (arc mark) actually generated. In the present embodiment, the management device 12 uses not only the image data of the abnormal part actually generated, but also a pseudo image (similar image) representing the abnormal / degraded state generated in the inspection object, using a neural network. Generated using learning (deep learning) technology. In this embodiment, using a deep learning technique, the ratio of the combination of the abnormal image (abnormal image) and the normal image around the abnormal region and the normal image and the abnormal image is changed. While learning similar images, a learning model based on a large number of similar images is learned (generated). Therefore, the learning model generated by the management device 12 is stored in the learning model 21a, and an abnormal part is searched from the image data using the learning model stored in the learning model 21a, so that it is surely generated in the inspection object. It is possible to detect abnormal points.

  The detection target instruction input unit 21b inputs an instruction about a detection target such as an inspection object included in the power transmission equipment 17 to be a patrol inspection target and an abnormality type to be a detection target in accordance with an input operation of the patrolman.

  The abnormal part detection unit 21c is detected by image processing based on the learning model (features of the abnormal part of the inspection object) stored in the learning model 21a from the acquisition data stored in the acquisition data storage unit 21d. An abnormal location corresponding to the instruction of the detection target input by the instruction input unit 21b is detected.

  The acquired data storage unit 21d stores acquired data (see FIG. 23) acquired from the drone 14. The acquired data includes, for example, moving image data obtained by photographing the inspection object (may be still image data), position data (latitude, longitude, altitude) indicating the position of the drone 14 at the time of image photographing, inspection object (tour / The inspection / inspection object data for discriminating the period (or start timing) of photographing for each inspection object) is included.

  The image confirmation unit 21e displays the acquired data stored in the acquired data storage unit 21d or an image corresponding to the processing result by the abnormal part detection unit 21c. The image confirmation unit 21e includes an abnormal part display unit 21f. The abnormal part display unit 21f displays an abnormal candidate image display screen (see FIG. 27) for displaying an image of a part (abnormal candidate image) as a candidate for the abnormal part detected from the image data by the abnormal part detection unit 21c. An abnormality location confirmation screen (see FIG. 28) for the operator to finally confirm the candidate abnormality candidate image is displayed.

  The abnormal part storage unit 21g stores abnormal part image data which is a candidate for the abnormal part detected from the image data by the abnormal part detection unit 21c. The abnormal part image data stored in the abnormal part storage unit 21g is read by the acquired data organization management unit 21 and displayed on the abnormal candidate image display screen or the abnormal part confirmation screen.

  The confirmation instruction input unit 21h inputs a confirmation instruction by a worker or the like to the abnormality candidate image display screen or the abnormal part confirmation screen. As for the abnormal part confirmation screen, as a result of the operator confirming the abnormal candidate image, it can be instructed whether it is an abnormal part (defect) or misrecognition (no abnormality). Abnormality types (for example, arc marks, wire breakage, charcoal adhesion, gloss from which charcoal has been peeled off) and the like can be indicated.

  The confirmation data storage unit 21k includes, on the abnormal part confirmation screen, image data (abnormal part image data) in which the abnormality candidate image is confirmed to be an abnormal part, an inspection object to be detected, an abnormality type to be detected, Each data indicating the detection position, detection date and time is stored as confirmation data. The confirmation data is used, for example, as inspection inspection data in the inspection inspection management unit 22.

  The abnormal part image data storage unit 21m stores abnormal part image data, an inspection object to be detected, and data indicating an abnormal type to be detected in order to be used for generating a learning model in the management device 12.

  The inspection inspection management unit 22 performs inspection / inspection management for collecting processing results for the data acquired from the drone 14, and includes an inspection inspection data storage unit 22a and an inspection inspection result report creation unit 22b.

  The inspection inspection data storage unit 22a stores inspection inspection data as a processing result by the acquired data organization management unit 21. The inspection inspection data includes, for example, the inspected object in which the abnormal part is detected, the abnormal part image data in which the abnormal part is confirmed, the abnormal type of the abnormal part, the position where the abnormal part is detected (the position in the inspected object, This includes the location (latitude, longitude, etc.) of the abnormal part in the inspection object.

  The inspection inspection result report creation unit 22b generates inspection inspection result report data that summarizes the inspection inspection results based on the inspection inspection data stored in the inspection inspection data storage unit 22a.

FIG. 5 is a diagram showing an example of the appearance of the drone 14 in the present embodiment.
The drone 14 shown in FIG. 5 has a drone main body 14a that accommodates a control unit and the like, four rotors 14b that are symmetrically provided upward from the drone main body 14a, and a shooting direction that is variable below the drone main body 14a. And a camera 14c provided as described above. The rotor 14b is a rotor blade that generates upward lift by rotation, and the rotation is controlled by the control unit, so that the drone 14 can be moved in the vertical direction and the horizontal direction. It is also possible to cause hovering to continue flying at substantially the same position. The camera 14c is a digital camera, a video camera, or the like that captures a still image or a moving image.

  FIG. 6 is a block diagram showing a main configuration of the drone 14 in the present embodiment. The drone body 14a of the drone 14 includes a control unit 30a, an acquisition data generation unit 30b, an imaging control unit 30c, a flight control unit 30d, a sensor group 30e, a memory 30f, a GPS data reception unit 30g, a wireless communication unit 30h, and a camera unit 30k. A drive unit 30m is provided.

  The control unit 30a governs overall control of the drone 14, and may be configured as a dedicated controller or may be a general-purpose unit including a processor and a memory. The control unit 30a includes functional modules of an acquisition data generation unit 30b, an imaging control unit 30c, and a flight control unit 30d that are realized by executing each control program.

  The acquisition data generation unit 30 b generates acquisition data to be provided to the information processing apparatus 10. The acquisition data generation unit 30b is, for example, image data of an image (moving image or still image) captured by the camera 14c, position data (latitude) indicating the position of the drone 14 at the time of image generation generated by the GPS data reception unit 30g. , Longitude, altitude), and patrol / inspection indicating the period (or start timing) of each object to be inspected (patrol / inspection object) based on flight control data used for flight control by the flight control unit 30d Acquired data is generated in association with the target data (see FIG. 23).

  The imaging control unit 30c performs control for imaging the power transmission equipment 17 to be a patrol inspection while the drone 14 is flying under the control of the flight control unit 30d. The imaging control unit 30c has functions of controlling the lens, shutter, aperture, focus mechanism, zoom mechanism, and the like of the camera 14c, and adjusting the imaging direction (lens direction), for example. As a function of adjusting the shooting direction, the camera unit 30k may be driven to change the shooting direction by rotating the lens direction of the camera 14c horizontally or vertically with respect to the drone body 14a. The drive of the drive unit 30m may be controlled through the unit 30d, and the drone 14 may be tilted to change the shooting direction of the camera 14c. The camera 14c can capture not only a downward image of the drone 14 but also a horizontal or upward image.

  The flight control unit 30d includes flight control data received from the information processing apparatus 10, position data indicating the current position of the drone 14 generated by the GPS data receiving unit 30g, obstacles detected by the sensor group 30e, etc. Based on the position of an object etc., the drive unit 30m (motor) is driven and the drone 14 is controlled to fly autonomously. When the flight control unit 30d acquires the flight control data, the flight control unit 30d acquires the position data from the GPS data receiving unit 30g, and controls the drone 14 to fly from the inspection object of the power transmission facility 17 within a predetermined range.

  The sensor group 30e acquires various information such as position, altitude, speed, and direction, for example, a three-axis accelerometer that measures the acceleration of an object, a gyroscope that detects an angle or angular velocity of an object, and a three-axis magnetic sensor. Etc. The sensor group 30e includes an obstacle camera for detecting an obstacle or an inspection object. For example, by providing a plurality (for example, four) of obstacle cameras, the flight control unit 30d three-dimensionally processes the images of the obstacles and the inspection object, and the positions of the obstacles and the inspection object are displayed in a three-dimensional space. Can be recognized. This enables flight control such as avoiding contact / collision with an obstacle or an inspected object or flying at a predetermined distance from the inspected object.

  The memory 30f stores various data. For example, flight control data received from the information processing apparatus 10, acquisition data generated by the acquisition data generation unit 30b, and the like are stored. The memory 30f can be, for example, a detachable and portable memory medium. When the acquired data is provided to the information processing apparatus 10, the memory 30f is removed from the drone 14 and causes the information processing apparatus 10 to read the data. You can also.

  The GPS data receiving unit 30g receives a GPS signal from the GPS satellite 16 and generates position data (latitude, longitude, altitude) and time data based on the current position of the drone 14 based on the GPS signal. The position data generated by the GPS data receiving unit 30g is used for autonomous flight control and acquisition data generation by the flight control unit 30d.

  The wireless communication unit 30h controls wireless communication with the information processing apparatus 10 or with a base station accommodated in a wireless public network. The wireless communication unit 30h may transmit acquired data (image data) captured during the flight of the drone 14 to the information processing apparatus 10 in real time under the control of the control unit 30a. In addition, the wireless communication unit 30h can receive flight control data corresponding to an operation performed by the operator on the information processing apparatus 10 during flight in order to fly the drone 14 by manual control.

  The camera unit 30k supports the camera 14c in the lower part of the drone body 14a and is driven by the control of the shooting control unit 30c to adjust the shooting direction of the camera 14c.

  The drive unit 30m is driven according to the control of the flight control unit 30d, and individually rotates the plurality of rotors 14b. By individually controlling the rotation of the plurality of rotors 14b, it is possible to adjust the posture such as moving the drone 14 in the vertical direction or the horizontal direction, or hovering or tilting the drone body 14a.

  Next, the management apparatus 12 in this embodiment will be described.

  For example, the management apparatus 12 has the same configuration as the information processing apparatus 10 shown in FIG. The management apparatus 12 implements various functions for generating a learning model used in image processing in the information processing apparatus 10 by executing a learning model generation program using a processor.

FIG. 7 is a block diagram illustrating a functional configuration realized by executing a learning model generation program by a processor in the management apparatus 12 according to the present embodiment.
Based on the learning model generation program, the management device 12 includes a data storage unit 40a (abnormal part image data 40b, learning image data 40c, teaching metadata 40d, verification image data 40e), learning image reading unit 40f, teaching metadata. Reading unit 40g, edge cutout unit 40h, adjacent normal image search unit 40k, normal image / abnormal image combination control unit 40m, similar image generation unit 40n, learned model management unit 40p, verification image reading unit 40q, inference execution unit 40r, The functions of the index / condition relation storage unit 40s, the relation correspondence unit 40t, the inquiry unit 40u, and the learning model storage unit 40v are realized.

  Next, learning model generation processing for generating a learning model by the management device 12 will be described. FIG. 8 is a flowchart for explaining the learning model generation process executed by the management device 12 in the present embodiment. In the following explanation, an explanation will be given by taking as an example a case where an aerial ground line included in the power transmission facility 17 or an arc mark generated on the power transmission line is set as an abnormal location and a learning model for detecting this abnormal location is generated. . Similarly, a learning model for detecting an abnormal portion occurring in another inspection object is similarly described, and a detailed description is omitted assuming that a learning model corresponding to the abnormal portion of each inspection object is generated.

  In this embodiment, since there are few samples of image data of arc traces caused by lightning strikes on overhead ground wires or power transmission lines, not only abnormal location image data of actually generated abnormal locations (arc traces) Deep learning (similar image) is generated based on image data (learning image data) of the abnormal / degraded state artificially added to the inspected object, and deep learning using a neural network ( A large number of learning models are learned (generated) using deep learning) technology.

  In the management device 12, data necessary for generating a learning model is stored in advance by the data storage unit 40a. The data stored by the data storage unit 40a includes abnormal part image data 40b, learning image data 40c, teaching metadata 40d, and verification image data 40e.

  The abnormal part image data 40b is actual image data of an abnormal part acquired by actually performing a patrol inspection on the inspection object using the information processing apparatus 10 and the drone 14. As the abnormal part image data 40b, image data in which an abnormal part and an abnormal type are confirmed by an operator or the like from image data photographed by the drone 14 is used.

  The learning image data 40c is learning pseudo image data obtained by photographing a pseudo-added abnormality / deterioration state. The learning image data 40c includes, for example, a state in which a strand is cut, a state in which the transmission line is charcoal, a state in which the charcoal is removed and the transmission line is shining (glossy), etc. It is generated by photographing this power transmission line.

  The teaching metadata 40d is data necessary to generate a similar image (learning model) that is instructed (taught) by an operation of a worker or the like with respect to the abnormal part image data 40b and the learning image data 40c. The teaching metadata 40d includes, for example, an area including an inspection object in an image, an area including a portion corresponding to an abnormal part (abnormal image), a type such as abnormality / deterioration (abnormal type), and an area to be distinguished from a detection target ( Data indicating a region not to be processed).

  The generation of the learning image data 40c and the teaching metadata 40d is executed in the management device 12 or the information processing device 10. For example, when it is executed in the information processing apparatus 10, an image is displayed based on the abnormal part image data 40b or the learning image data 40c, and an instruction (teaching) for an area corresponding to this image is made according to the operation of a worker or the like Enter.

  For example, when the inspection object is a power transmission line, an operator or the like operates a pointing device or the like to set a range including the entire power transmission line in the image as a region of the inspection object. (Positions indicating the two vertices) of an abnormal position such as an arc mark on the power transmission line (abnormal image) is taught in a rectangular area. Further, the worker or the like instructs the addition of the arc mark label as information indicating the type of abnormality such as abnormality / deterioration. In addition, if there is an area that you want to clearly distinguish from the arc mark at the abnormal location, such as a hard snow ring attached to a power transmission line, workers etc. can teach the equivalent area to the hard snow ring in the same way. it can.

  The information processing apparatus 10 generates teaching metadata 40d according to an instruction (teaching) input by an operation of a worker or the like, and creates a file associated with each of the abnormal part image data 40b or the learning image data 40c. Remember.

  The verification image data 40e is verification image data for evaluating the recognition accuracy of the abnormal part using the learning model generated based on the abnormal part image data 40b, the learning image data 40c, and the teaching metadata 40d. is there. The verification image data 40e includes, for example, image data for a plurality of sheets.

  Hereinafter, a case where a learning model is generated using the learning image data 40c will be described as an example. It should be noted that when only the learning image data 40c is used, the learning image data 40c and the abnormal part image data 40b are mixedly used, and further, only the abnormal part image data 40b is used. Anyway.

  First, the learning image reading unit 40f reads the learning image data 40c stored in advance in the data storage unit 40a (step A1). The teaching metadata reading unit 40g reads teaching metadata 40d corresponding to the learning image data 40c read by the learning image reading unit 40f.

  The edge cutout unit 40h uses the teaching metadata 40d read by the teaching metadata reading unit 40g for the learning image data 40c read by the learning image reading unit 40f, for example, an image processing technique for edge detection. Is used to cut out the outline of the power transmission line (step A2). That is, the edge cutout unit 40h cuts out a region corresponding to a power transmission line from a region including the entire power transmission line in the learning image indicated by the teaching metadata 40d.

  Note that the edge cutout unit 40h not only detects the edge of the entire outline of the region corresponding to the power transmission line, but can also cut out the region corresponding to the power transmission line by simpler processing as described below. A range including the entire power transmission line is taught by a rectangular area. The image of the power transmission line included in this rectangular area exists by lowering to the right with the upper left as one of the vertices or rising to the right with the lower left as one of the vertices. The contour of the image of the power transmission line can be approximated by a straight line when viewed locally. Therefore, for example, by detecting edges at at least two locations separated in the left-right direction, it is determined whether the image of the power transmission line included in the rectangular area is right-down or right-up based on the positions of the edges. can do. A straight line passing through the edge position detected in the left-right direction can be regarded as the contour of the approximated power transmission line image. As a result, it is not necessary to detect the edge of the entire outline of the power transmission line area, so that the processing load can be reduced.

  Next, the adjacent normal image search unit 40k sets a region corresponding to the abnormal portion indicated by the teaching metadata 40d, that is, a portion of the abnormal portion (abnormal image region) such as an arc mark (step A3), and sets the abnormal image region. A plurality of normal image regions around the abnormal image in the image of the inspection object (within the outline of the power transmission line) are searched (step A4).

  For example, the adjacent normal image search unit 40k searches for a normal image area using a rectangular area having the same size as the rectangular area indicating the abnormal image area indicated by the teaching metadata 40d. Here, the adjacent normal image search unit 40k uses, for example, a region excluding a region to be distinguished from the detection target indicated by the teaching metadata 40d (a region not to be processed such as a difficult snow ring) as the next priority. A plurality of normal image areas are searched based on the degrees (1) to (4).

  (1) a normal image area closest to either the right or left of the rectangular area of the abnormal image (usually adjacent), (2) a normal image area closest to (normally adjacent to) one of the upper or lower of the rectangular area of the abnormal image (3) A normal image region located in any of the upper left, upper right, lower left, and lower right of the rectangular region of the abnormal image, and (4) a normal image closest to the normal image region that has been searched first (usually adjacent) region.

  9, 10, and 11 show an example of the abnormal image area 56 indicated by the teaching metadata 40 d and a plurality of normal image areas 58 searched based on the position of the abnormal image area 56. Yes. As shown in FIGS. 9 to 11, a region 52 including a power transmission line 50 in an image, a region 56 including a portion corresponding to an abnormal part (abnormal image), and a region 54 (difficult snow ring region) to be distinguished from a detection target. ) Is indicated by the teaching metadata 40d. Note that the area 56 may be instructed by one rectangular area with respect to one abnormal place, or may be instructed by adjoining a plurality of rectangular areas. 9, FIG. 10 and FIG. 11 show examples in which the area 56 is designated (taught) by a square. In the example shown in FIG. 9, one abnormal location is indicated by two areas 56. In the example shown in FIG. 10, the same four areas 56 are indicated, and in the example shown in FIG. 11, the same area 56 is indicated.

  In FIG. 9, the normal image areas 58 adjacent to the left and right of the area 56 are searched according to the priorities described above, and the searched area 58 and the normal image areas 58 adjacent to the left and right are searched. A region 58 adjacent to the left and right is searched. In FIG. 9, the area 54 (difficult snow ring) to be distinguished from the detection target is indicated. The area 58 is searched with the area 54 excluded from the search target.

FIG. 10 shows an example in which a normal image area 58 located in the lower left or upper right of the abnormal image area 56 is searched according to the above-described priorities. In FIG. 11, the normal image area 58 is searched to the left and below the abnormal image area 56 according to the above-described priorities, and the normal image area 58 adjacent to the left of the searched area 58 is searched. .
In the example shown in FIGS. 9 to 11, the areas 56 and 58 are described as squares, but may be rectangular. The adjacent normal image search unit 40k may extract all normal images that can be extracted from the image of the inspection object (power transmission line 50), or is necessary for the processing of the normal image / abnormal image combination control unit 40m. It is good also as the number of sheets decided beforehand.

  Next, the normal image / abnormal image combination control unit 40m sets a ratio of combining the abnormal image in the region 56 indicated by the teaching metadata 40d and the abnormal images in the plurality of regions 58 searched by the region adjacent normal image search unit 40k. Set (step A5). For example, the normal image / abnormal image combination control unit 40m changes the ratio of combining the normal image and the abnormal image stepwise, for example, from 1: 1 to 10: 1.

  The similar image generation unit 40n uses a technique of deep learning (deep learning) utilizing a neural network, and according to the ratio of combining the normal image and the abnormal image set by the normal image / abnormal image combination control unit 40m, A plurality of normal images searched by the adjacent normal image search unit 40k and an abnormal image indicated by the teaching metadata 40d are combined (step A6), and a pseudo image (similar image) representing the state of abnormality / deterioration occurring in the inspection object Is generated (step A7). The pseudo image combining the normal image and the abnormal image can be generated using, for example, an existing image processing method.

  For example, when the ratio of combining a normal image and an abnormal image is 1: 1, one normal image is selected from a plurality of normal images and combined for one abnormal image. A plurality of pseudo images can be generated by changing a target to be selected from a plurality of normal images and combining them with abnormal images.

  In addition to combining the abnormal image and the normal image included in the learning pseudo image indicated by the single learning image data 40c, the abnormal image and the normal image included in the plurality of other learning pseudo images are also combined. It can also be combined. Also, a combination of a plurality of normal images and a plurality of abnormal images is possible as long as the ratio of normal images and abnormal images is matched. For example, when the ratio of combining a normal image and an abnormal image is 1: 1, it is possible to combine two normal images and two abnormal images. In this case, the combination of two normal images can be changed from three or more normal images. Similarly, the combination of two abnormal images can be changed from three or more abnormal images. . Therefore, the number of combinations of a plurality of normal images and a plurality of abnormal images can be increased by changing the combination of the normal images and the abnormal images.

  FIG. 12 conceptually shows a combination of an abnormal image and a normal image by the normal image / abnormal image combination control unit 40m in the present embodiment. As described above, a large number of similar images can be generated by combining a plurality of abnormal images (A) and a plurality of normal images (B).

  As described above, the similar image generation unit 40n uses the normal image / abnormal image combination control unit 40m to change the normal image and the abnormal image step by step by deep learning. Are combined to generate a similar image (steps A5 to A7).

  The learned model management unit 40p learns similar images according to the ratio of the normal image and the abnormal image generated by the similar image generation unit 40n (step A9). That is, the learned model management unit 40p associates a similar image generated by deep learning with a condition for generating a similar image, for example, a condition (generated image condition) such as a ratio of an abnormal image and a normal image. And storing / managing in the learning model storage unit 40v.

  In the above-described processing for generating a series of similar images, similar images (learning models) corresponding to the types of abnormalities occurring in the inspection object are generated. For example, regarding arc traces generated in an overhead ground wire or a power transmission line, similar images are individually generated and learned for each of the subdivided categories such as wire breakage, charcoal adhesion, and luster from which charcoal is peeled off.

  When a similar image (learning model) is learned by the learned model management unit 40p, the inference execution unit 40r causes the verification image reading unit 40q to read the verification image data 40e from the data storage unit 40a (step A10), thereby verifying the verification image. An index representing the accuracy (recognition accuracy) of authentication using the learning model for the data 40e (verification image) is inferred (step A11). That is, the learned model management unit 40p is generated for an abnormality type of an abnormal part included in the verification image, for example, for a plurality of verification images in which an image including an abnormal part and an image including no abnormal part are mixed. The abnormal part detection (authentication) process is executed using the learned model. Here, the inference execution unit 40r executes an abnormal location detection (authentication) process using each of the learning models having different generation image conditions (a ratio of the abnormal image and the normal image combined) that generated the similar image. Thus, an index of authentication accuracy for each generated image condition is inferred.

  The inference execution unit 40r obtains, for example, a “recall rate” and a “relevance rate” as indexes representing the recognition accuracy. Other indices may be obtained.

  The “reproducibility” indicates a ratio of not overlooking an abnormal part generated in the inspection object (such as a power transmission line), and has a meaning of a ratio of reproducing the abnormal part. For example, when an abnormal part is detected (recognized) from 20 verification images as a result of executing an abnormal part detection (authentication) process using a learning model for a verification image including 25 abnormal images. Is a reproduction rate (a rate not to be missed) of 80%.

  “Accuracy rate” indicates the percentage of parts that were actually abnormal as a result of detecting abnormal parts that occurred on the inspection object (such as power transmission lines) (does not shake). It means the conforming ratio. For example, when an abnormal part is detected from 20 verification images as a result of executing an abnormal part detection (authentication) process using a learning model, when there are actually 15 verification images including the abnormal part The matching rate (the rate of not swinging) is 75% (in this case, 5 sheets (25%) are skipped).

  The index / condition-related storage unit 40s generates the index (“reproduction rate” “matching rate”) obtained by the inference execution unit 40r and the generated image condition that generates the learning model (similar image) used when obtaining this index. The relationship with the (ratio combining the abnormal image and the normal image) is stored (step A12).

  FIG. 13 is a diagram showing the relationship between the generated image condition stored in the index / condition relation storage unit 40s and the recall ratio, and FIG. 14 is compatible with the generated image condition stored in the index / condition relation storage unit 40s. It is a figure which shows the relationship of a rate. FIGS. 13 and 14 show index values for the categories of wire breakage (Cut), charcoal adhesion (Chacoal), and gloss (Glare) from which charcoal has been peeled as abnormal types of arc marks.

  The inquiry unit 40u, for example, according to the operation of a worker or the like, the abnormality type of the abnormal part to be detected (for example, arc trace of the transmission line, wire breakage, charcoal adhesion, gloss, etc.), the worker, etc. Enter at least one of the recall ratio and precision ratio. The inquiry unit 40u learns to obtain, for example, the best detection result of the abnormal part for the related corresponding part 40t in accordance with the designation of the abnormality type of the input abnormal part and at least one of the reproduction rate or the matching rate. Queries the model (generated image condition).

  In response to the inquiry (query) from the inquiry unit 40u, the related correspondence unit 40t can obtain the best detection result of the abnormal part based on the index and the generated image condition stored in the index / condition related storage unit 40s. The learning model (generated image condition) is responded and presented to a worker or the like.

  For example, it is assumed that “general arc mark” and “both reproduction rate / matching rate” are specified and an inquiry “query is the best generated image condition?” Is input. In this case, when the index and the generation image condition shown in FIGS. 13 and 14 are stored, the related correspondence unit 40t reproduces each of the wire breakage (Cut), charcoal adhesion (Chacoal), and gloss (Glare). Using the relevance ratio, a generated image condition (ratio of normal image to abnormal image) that obtains the best numerical value by a predetermined calculation method is obtained. As a result, it is required that the best result is obtained when the ratio of combining the normal image and the abnormal image is 8: 1. That is, in the abnormal part detection process (image processing for detecting an abnormal part) of the information processing apparatus 10, when an abnormal part is detected for the “general arc mark” from the image of the inspection object, the ratio between the normal image and the abnormal image is When a learning model corresponding to 8: 1 is used, it is possible to detect an abnormal place best.

  In addition, it is assumed that “wire breakage” and “both reproduction rate / relevance rate” are specified, and an inquiry (query) “What is the best generated image condition?” Is input. In this case, the related correspondence unit 40t uses the reproduction rate and the matching rate of the broken wire to determine the generated image condition (the ratio between the normal image and the abnormal image) that provides the best numerical value by a predetermined calculation method. Ask. As a result, it is required that the best result is obtained when the ratio of the normal image to the abnormal image is 9: 1. That is, in the abnormal part detection process of the information processing apparatus 10, when detecting an abnormal part with respect to “wire breakage” from the image of the inspection object, a learning model in which the ratio of the normal image to the abnormal image corresponds to 9: 1 is selected. When used, it is possible to detect an abnormal point best.

  Thus, based on the result of the inquiry from the inquiry unit 40u, the learning model used for the abnormal part detection process in the information processing apparatus 10 can be determined and stored in the information processing apparatus 10 (learning model 21a). For example, in the information processing apparatus 10, when “abnormal arc mark”, “wire breakage”, “charcoal adhesion”, and “gloss” are individually selected as the abnormality types so that the abnormal part can be detected, the respective cases are handled. A learning model is determined. In addition, when the information processing apparatus 10 selects either “reproduction rate” or “relevance rate” so that an abnormal part can be detected, either “reproduction rate” or “relevance rate” is selected. Based on this, a learning model is determined.

  In the above description, the generated image condition corresponding to the inquiry is returned by making an inquiry from the inquiry unit 40u in response to an instruction input by an operation of a worker or the like. Alternatively, a learning model condition that satisfies this condition may be transmitted to the information processing apparatus 10 by setting conditions for the learning model used in the information processing apparatus 10 in advance. The learning model condition can be the same as that of the above-described inquiry (designation of at least one of the abnormality type, the recall rate, or the matching rate).

  In the above description, one “best generated image condition” is determined and used as a learning model to be used in the abnormal part detection process of the information processing apparatus 10. For example, a plurality of generated image conditions (for example, normal images) / The learning image with an abnormal image of 9: 1 and 8: 1) may be combined and used in the information processing apparatus 10.

  As described above, the management device 12 according to the present embodiment uses the deep learning (deep learning) technique using the abnormal image to be detected such as abnormality / deterioration and the normal image in the vicinity of the abnormal image. A similar image can be automatically generated efficiently and accurately, and a learning model used for abnormal part detection processing in the information processing apparatus 10 can be generated.

  In addition, although the process which produces | generates the learning model in the management apparatus 12 mentioned above is demonstrated taking the example of the abnormal location (arc trace etc. of a power transmission line) which generate | occur | produces in the power transmission equipment 17, it is an industrial apparatus / structure, etc. Therefore, the present invention can be widely applied to systems / apparatuses that discriminate abnormalities / deteriorations by image recognition.

Next, the inspection inspection operation by the information processing apparatus 10 and the drone 14 in the present embodiment will be described. Here, for example, it is assumed that an abnormal portion due to a lightning strike occurs in the power transmission facility 17.
For example, when a worker or the like performs a patrol inspection using the drone 14 on the power transmission facility 17, the information processing apparatus 10 and the drone 14 are mounted on an automobile or the like, and the drone 14 can fly to the power transmission facility 17. Move within a safe range.

  First, in the information processing apparatus 10, a flight plan is set according to the power transmission equipment 17 to be subject to inspection inspection. The information processing apparatus 10 displays a patrol / inspection route selection screen 60 for setting a flight plan as shown in FIG. 15, for example, and is selected by a worker or the like. On the patrol / inspection route selection screen 60, for example, a regular patrol button 61, an accident patrol button 62, and a tower patrol button 63 are provided, and any of these can be arbitrarily selected. The accident patrol is carried out, for example, when an accident such as a lightning strike to the power transmission facility 17 occurs to identify the cause of the accident. Regular patrols and tower patrols are patrols for regular patrols or inspections.

  When an accident patrol is selected by the accident patrol button 62, the flight plan processing unit 20b of the information processing apparatus 10 inputs the selection contents by the patrol inspection instruction input unit 20a, and, for example, as shown in FIG. A flight plan (flight control data) is generated by photographing the inspected objects (flashlight indicator, arc horn, insulator, etc.) and images of the overhead ground line and the power transmission line. In addition, when the regular tour is selected by the regular tour button 61, the flight plan processing unit 20b similarly takes a flight plan (flight that can photograph, for example, an image of an overhead ground line and a power transmission line as shown in FIG. Control data). Although not shown, when the tower patrol button 63 selects the tower patrol, the flight plan processing unit 20b can take a flight plan (flight control data) that can capture an image of the target tower from the lower part to the upper end. Is generated.

  In addition, about the power transmission equipment 17 which is the object of the patrol inspection, basic information such as the position where the steel tower is installed and the shape of the steel tower are stored in the information processing apparatus 10 and can be designated by a worker or the like. The flight plan processing unit 20 b generates flight control data based on basic information such as the position of the power transmission equipment 17 to be subjected to the patrol inspection and the flight plan of the patrol inspection selected on the patrol / inspection route selection screen 60.

  In the patrol / inspection route selection screen 60, normal buttons 61a, 62a, 63a and winter buttons 61b, 62b, 63b are provided for the regular patrol button 61, the accident patrol button 62, and the tower patrol button 63, respectively. ing.

  Generally, lightning strikes are discharged downward from above the power transmission facility 17, and for example, when a lightning strike occurs on an overhead ground wire, an abnormal portion such as an arc mark occurs on the upper surface side of the overhead ground wire. Therefore, by selecting the normal buttons 61a, 62a, and 63a (default), a flight plan is created in which the drone 14 can capture an image above the imaginary ground line.

  On the other hand, during the winter season in the Hokuriku region, there may be winter thunders that are discharged upward from the ground surface. Therefore, when a lightning strike occurs in the power transmission facility 17 due to winter lightning, there is a possibility that an abnormal portion such as an arc mark may occur on the lower surface side of the transmission line. Therefore, it is possible to create a flight plan that can capture an image of the lower surface side of the overhead ground wire or the power transmission line by selecting the winter button 61b, 62b, 63b.

  16 and 17 show an example in which an image including an aerial ground line and a plurality of power transmission lines is taken by flying the drone 14 over the power transmission facility 17, for example, In order to individually capture images of a plurality of power transmission lines, it may be caused to fly so as to reciprocate between steel towers. Also, in order to take images of abnormal places other than lightning strikes, for example, abnormal places caused by contact of cranes, flying objects, trees, etc. from the side of the transmission line, images are taken from the side of the transmission line. You may fly.

  18, 19, and 20 illustrate an example of an inspection object provided in the power transmission facility 17. FIG. 18 shows one steel tower 17a. An overhead ground wire 17b and three power transmission lines 17c, 17d, and 17e are usually installed on the steel tower 17a. The power transmission lines 17c, 17d, and 17e are respectively installed on the steel tower 17a via insulators 17h. FIG. 19 shows an example of the insulator 17h. The insulator 17h is provided with an arc horn 17g to prevent the insulator 17h from being destroyed when a lightning strike occurs on the steel tower 17a. The steel tower 17a is equipped with a flashing indicator 17f for notifying that a lightning strike has occurred when there is a lightning strike on the steel tower 17a or the overhead ground wire 17b. FIG. 20 shows an example of the flashing indicator 17f. FIG. 20A shows the state of the flashing indicator 17f in a normal state, and FIG. 20B shows the flashing indicator 17f when a lightning strike occurs on the steel tower 17a. As shown in FIG. 20 (B), when a lightning strike occurs on the tower 17a or the overhead ground wire 17b, the flashing indicator 17f opens the lid 17f2 from the display body 17f1 by the current flowing through the tower 17a, and is housed inside. The displayed display cloth 17f3 is exposed.

  In the management device 12, the learning model for the arc horn 17g, the insulator 17h, and the flashing indicator 17f is created in the same manner as the learning model for the abnormal state occurring in the above-described transmission line, so that these inspection objects are also It can be detected from an image photographed by the drone 14. For example, for the arc horn 17g, a learning model capable of detecting an arc mark caused by a lightning strike is generated. For the insulator 17h, a learning model capable of detecting damage (cracks) caused by lightning is generated.

  FIG. 21 shows an example of a flight plan (flight control data) generated by the flight plan processing unit 20b in the present embodiment. The flight control data is for flying a series of flight routes for photographing the power transmission equipment 17 to be inspected for inspection by the drone 14, and for example, the inspection / inspection target data classified for each inspection / inspection object corresponds. It is attached.

Next, the operation of the drone 14 in this embodiment will be described with reference to the flowchart shown in FIG.
The control unit 30a of the drone 14 stores the flight control data received from the information processing apparatus 10, and when the flight start is instructed, the flight control unit 30d drives the drive unit 30m to respond to the flight control data. Flight control for flying the route is started (step B1).

  In addition, the control unit 30a generates and stores acquired data by the acquired data generation unit 30b. In other words, the acquired data generation unit 30b stores the tour / inspection target data indicating the period (or start timing) in which each subject to be inspected (tour / inspection object) was imaged (step B2), and is captured by the camera 14c. Recording of image data (here, for example, moving image data) and storage of position data generated by the GPS data receiving unit 30g are started (steps B2, B3, B4).

  FIG. 23 shows an example of acquisition data generated by the acquisition data generation unit 30b in the present embodiment. As shown in FIG. 23, the acquisition data generation unit 30b includes moving image data captured by the camera 14c (FIG. 23A), position data generated by the GPS data reception unit 30g (FIG. 23C), In addition, the traveling / inspection target data (FIG. 23B) is stored in association with each other.

  Since the first patrol / inspection object is “steel tower B-1”, “steel tower B-1” is stored as the patrol / inspection object data. Moving image data and position data are continuously stored. The position data may be stored every predetermined time. Still image data may be stored continuously instead of moving image data.

  As shown in FIG. 23, by storing the tour / inspection data, after the photographing by the drone 14 is completed, the recording range for each inspection object included in the moving image data based on the tour / inspection data. Can be easily searched. For example, the range M2 of the moving image data in which the “imaginary ground line” is photographed can be searched based on the traveling / inspection target data indicating the “imaginary ground line”. Further, since the moving image data and the position data are stored in association with each other, for example, when an abnormal part is captured in the range M2 of the moving image data, the position of the drone 14 when the abnormal part is captured is shown. Based on the position data PE, it is possible to roughly grasp the position (latitude, longitude) of the abnormal part occurring in the “imaginary ground wire”.

  In the above description, the acquired data is stored in the drone 14 (for example, the memory 30f), but can be transmitted to the information processing apparatus 10 by wireless communication in real time (step B5).

  When the flight in the range corresponding to the patrol / inspection object is completed (Yes in Step B6), the acquired data generation unit 30b does not complete the flight in the range corresponding to all the patrol / inspection objects (No in Step B7). ) When the patrol / inspection object moves to the next inspection object, new patrol / inspection object data is stored (step B2). Thereafter, similarly, the moving image data and position data taken for the next patrol / inspection object are stored (steps B3 and B4).

  In the same manner, the recording of the acquired data is continued, and when the flight in the range corresponding to all the tours / inspections is completed (step B7, Yes), the control unit 30a returns the drone 14 to the original flight start position. Flight control is started and storage of acquired data is terminated.

Next, acquired data management processing in the information processing apparatus 10 according to the present embodiment will be described with reference to the flowcharts shown in FIGS.
The acquired data stored by the drone 14 is transmitted to the information processing apparatus 10 and stored in the acquired data storage unit 21d. The acquisition data storage unit 21d can store acquisition data stored by a plurality of flights by the drone 14. The information processing apparatus 10 starts organizing acquired data for detecting and confirming an abnormal part generated in the inspection object based on the image data by an operation of an operator or the like.

  First, the information processing apparatus 10 inputs designation of a tour / inspection route to be processed through the tour / inspection route selection screen 60, and selects a processing target from acquired data stored by the flight of the designated tour / inspection route. (Step C1). Further, the information processing apparatus 10 displays an inspection object item selection screen 70 for selecting an inspection object to be inspected, for example, as shown in FIG. Let them choose. The inspection item selection screen 70 is provided with, for example, an arc mark button 72, a strand break button 73, a charcoal adhesion button 74, and a gloss button 75 for all overhead ground wires and power transmission lines in addition to the button 71. Yes. The inspection object item selection screen 70 is provided with selection object items (not shown) for designating other inspection objects (insulators, arc horns, flashing indicators).

  For example, when the All button 71 is selected, the abnormal part detection unit 21c sets all the inspection objects photographed by the drone 14 in the designated patrol / inspection route as inspection targets, and each of the inspection objects. A learning model for detection is set (step C3).

  Similarly, for example, when the arc mark button 72 is selected, the abnormal point detection unit 21c detects the arc mark (including wire breakage, charcoal adhesion, and gloss categories) as a detection target, and the arc mark (wire breakage). , Charcoal adhesion, gloss) is set (step C3). Similarly, when any one of the strand break button 73, the charcoal adhesion button 74, and the gloss button 75 is selected, a learning model for detecting the corresponding abnormal part is set. As described above, the learning model for detecting the abnormal part is combined with the ratio (ratio) of the combination of the normal image and the abnormal image that can best detect the abnormal part for each abnormality type in the management device 12. It is generated based on similar images.

  Since the inspection target item can be selected on the inspection target item selection screen 70, the detection process of the abnormal part can be executed using the learning model corresponding to the specific abnormal part. Time can be shortened.

  After the setting of the inspection target item (learning model) is completed, the abnormal part detection unit 21c reads the acquisition data to be processed stored in the acquisition data storage unit 21d, and the circulation / inspection target data included in the acquisition data is read. It discriminate | determines (step C4). Here, the abnormal part detection unit 21c determines whether the traveling / inspection target data corresponds to the detection target of the abnormal part specified by the inspection target item.

  For example, when the arc mark button 72 for the overhead ground wire and the power transmission line is selected as the inspection target item on the inspection target item selection screen 70, the inspection / inspection target data may be “steel tower B-1”. For example, it is determined that it is not a detection target of an abnormal part. If the abnormal part detection unit 21c determines that it is not a detection target of the abnormal part (step C5, No), the moving image data is skipped to the traveling / inspection target data (for example, “aerial ground wire”) that is the detection target of the abnormality detection. Thus, moving image data obtained by photographing the detection target of the abnormal part is set as a processing target. That is, according to the inspection object item selected on the inspection object item selection screen 70, the processing time can be shortened by omitting the processing of the moving image data obtained by photographing the inspection object that is not the detection target of the abnormal part. Plan.

  The abnormal part detection unit 21c executes an abnormal part detection process for detecting an abnormal part as a detection target using the learning model set according to the inspection target item for moving image data in which the detection target of the abnormal part is captured. (Step C7).

FIG. 26 is a flowchart showing an abnormal point detection process in the present embodiment.
The abnormal part detection unit 21c detects a region corresponding to the inspection object from the moving image data of the acquired data stored in the acquired data storage unit 21d, and detects an abnormal image candidate region from the region. That is, a portion having a possibility of an abnormal portion in a state (shape, color, etc.) different from the surroundings in the inspection object is detected. The abnormal part detection unit 21c determines whether the image data of the abnormal image candidate region is an image corresponding to the abnormal type to be detected using a learning model that indicates the characteristics of the abnormal type that occurs in the inspection object ( Step D2).

  Here, when it is determined that the image represents an abnormal type (step D3, Yes), the abnormal part detection unit 21c determines that the image data (abnormal part image data) is determined to be an image corresponding to the abnormal type (abnormal image). ) And position data corresponding to the abnormal part image data is acquired. The abnormal part detection unit 21c stores abnormality type data indicating whether an abnormality is detected based on the learning model corresponding to which abnormality type, that is, which abnormality type is the above part. For example, when the arc mark button 72 is selected on the inspection target item selection screen 70, the determination of an abnormal point is performed using a learning model corresponding to each category of wire breakage, charcoal adhesion, and gloss, When the learning model corresponding to the broken wire is determined to be an abnormal part, abnormal type data indicating “broken wire” is stored.

  When the abnormal part is detected by the abnormal part detection process, the abnormal part detection unit 21c stores the data of the abnormal part in the abnormal part storage unit 21g (step D5). The abnormal part data includes, for example, abnormal part image data, position data (and time) corresponding to the abnormal part image data, abnormality type data, and traveling / inspection target data corresponding to a range of moving image data including the abnormal part image data. , Including a patrol / inspection route corresponding to moving image data.

  The abnormal part detection unit 21c performs the above-described processing until the acquired data to be processed ends (steps C4 to C13). In addition, while the process which detects an abnormal location is performed, when the abnormal location made into a detection target is changed through the inspection object item selection screen 70 by the instruction | indication from a worker etc., (Step C10, Yes) ), The abnormal part detection unit 21c resets the learning model according to the abnormal part (inspection target item) to be detected after the change, and restarts the process of detecting the abnormal part in the same manner as described above.

  On the other hand, the abnormal part display unit 21f displays an abnormal candidate image display screen for allowing a worker or the like to check the detection result based on the data of the abnormal part stored in the abnormal part storage unit 21g (step C9). .

FIG. 27 is a diagram showing an example of the abnormal candidate image display screen 80 in the present embodiment.
On the abnormality candidate image display screen 80, for example, the abnormality type 82 indicated by the abnormality type data is displayed together with the abnormal part image 81 (abnormality candidate image) based on the abnormal part image data. Further, reference information 83 related to the abnormal part image 81 is displayed on the abnormality candidate image display screen 80. Reference information 83 includes, for example, “Point B-1” indicating a patrol / inspection route, a shooting date and time, a detection position (latitude, longitude, altitude) indicated by position data, and an inspection object “upper transmission” indicated by patrol / inspection target data. "Electric wire" and "upper part" (indicating that the photograph is taken from the upper part of the transmission line). The reference information 83 is provided with a map button 84. When the map button 84 is selected, the abnormal location display unit 21f displays a map including the detection position (adds the position of the power transmission equipment 17 and the like), and maps the location where the abnormal location of the power transmission equipment 17 is detected. Make sure you can check above.

  When the abnormal part (candidate) is detected from the moving image data by the abnormal part storage unit 21g, the abnormal part display unit 21f displays each piece of information regarding the newly detected abnormal part (candidate) on the abnormal candidate image display screen 80. As described above, additional display is performed sequentially. Therefore, the worker or the like can check the detection contents at the time when the abnormal part is detected while executing the process of detecting the abnormal part on the acquired data.

  In addition, when an operation for selecting, for example, the abnormal part image 81 is performed on the abnormality candidate image display screen 80, the abnormal part display unit 21f finally confirms the abnormal part corresponding to the selected abnormal part image 81 by a worker or the like. To display the screen for confirming the abnormal location.

FIG. 28 is a diagram showing an example of the abnormal location confirmation screen 90 in the present embodiment.
The abnormal part confirmation screen 90 includes an abnormal part image 91 obtained by enlarging the abnormal part image 81 selected on the abnormal candidate image display screen 80, and an abnormal part obtained by reducing the abnormal part image that is not displayed on the abnormal candidate image display screen 80. Images 92a, 92b, and 92c are displayed.

  In addition, on the abnormal part confirmation screen 90, reference information 93 related to the abnormal part image 91 is displayed. The reference information 93 includes, for example, “Point B-1” indicating a patrol / inspection route, a shooting date and time, and a detection position (latitude, longitude, altitude). Further, the abnormal location confirmation screen 90 includes an abnormal location confirmation button 94 for inputting an abnormality type confirmation instruction by an operator, etc., and an erroneous recognition for inputting an instruction for confirming that the location is not an abnormal location by an operator. A button 95 is provided. The abnormality location confirmation button 94 is provided with buttons corresponding to each of a plurality of abnormality types for which confirmation instructions can be input, for example, an arc mark button 94a, a strand break button 94b, a charcoal adhesion button 94c, and a gloss button 94d. Yes.

  In the abnormal part confirmation screen 90, when any one of the direction buttons provided on the left and right sides of the abnormal part image 91 or the abnormal part images 92a, 92b, and 92c is selected, the abnormal part image of another abnormal part is displayed. 91 is displayed, and the abnormal part to be confirmed can be changed.

  As a result of referring to the abnormal part image 91 displayed on the abnormal part confirmation screen 90, for example, when it is confirmed that the element is broken, the worker or the like selects the broken line button 94b. When an abnormality confirmation instruction is selected by selecting the wire break button 94b (Yes in Step C11), the confirmation instruction input unit 21h confirms each data indicating the inspection object, abnormality type, detection position, detection date and time, etc. Is stored in the confirmation data storage unit 21k (step C12). The confirmation data stored in the confirmation data storage unit 21k is used, for example, as inspection inspection data for creating inspection inspection result report data in which inspection inspection results are summarized in the inspection inspection management unit 22.

  In addition, the confirmation instruction input unit 21h receives, in the management device 12, the same data as the confirmation data stored in the confirmation data storage unit 21k, the abnormal part image data, the inspection object to be detected, and the abnormality type data to be detected. In order to use it for the production | generation of a learning model, it stores in the abnormal location image data storage part 21m. The data stored in the abnormal part image data storage unit 21m is transmitted to the management device 12 at an arbitrary timing.

  In addition, as a result of referring to the abnormal part image 91, when it is confirmed that the part is not an abnormal part, the erroneous recognition button 95 is selected by an operator or the like. In this case, the confirmation instruction input unit 21h deletes the corresponding abnormal part image data from the abnormal part storage unit 21g.

  In this way, in the information processing apparatus 10, a learning model that can satisfactorily detect an abnormal part that is a detection target that occurs in the inspection object is set according to the inspection target item selected on the inspection target item selection screen 70. . Since the learning model is generated in the management device 12 using a deep learning technique including a pseudo image (similar image), even if there are few image samples of actual abnormal locations, It can be detected stably by image processing.

  In addition, the abnormal part image detected as the abnormal part candidate by the abnormal part detection process is referred to on the abnormal candidate image display screen 80 or the abnormal part confirmation screen 90 to determine whether the abnormal part actually occurs or not. It can be determined by such judgment. The abnormal part image data determined to be an abnormal part using the abnormal part confirmation button 94 on the abnormal part confirmation screen 90 is used to generate a learning model and is reflected in the learning model corresponding to the type of abnormality to be detected. Can be made.

  Further, the data of the abnormal part determined by using the abnormal part confirmation button 94 on the abnormal part confirmation screen 90 is used as inspection inspection data in the inspection inspection management unit 22, and the inspection inspection result report summarizing the results of the inspection inspection. Data can be created. Thereby, the work burden of the worker etc. who records the result of the inspection inspection can be reduced.

  Further, the confirmation of the abnormal part on the abnormal candidate image display screen 80 or the abnormal candidate image display screen 80 can be executed in parallel with the abnormal part detection process, so that the confirmation work can be shortened.

  The method described in each of the above embodiments is a program that can be executed by a computer, such as a magnetic disk (such as a hard disk), an optical disk (such as a CD-ROM or DVD), a magneto-optical disk (MO), or a semiconductor memory. It can also be stored in a storage medium and distributed. In addition, as long as the storage medium can store a program and can be read by a computer, the storage format may be any form.

  In addition, an OS (operating system) running on a computer based on an instruction of a program installed in the computer from a storage medium, MW (middleware) such as database management software, network software, and the like realize the above-described embodiment. A part of each process may be executed.

  Furthermore, the storage medium in each embodiment is not limited to a medium independent of a computer, but also includes a storage medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

  Further, the number of storage media is not limited to one, and the case where the processing in each of the above embodiments is executed from a plurality of media is also included in the storage media in the present invention, and the media configuration may be any configuration.

  The computer in each embodiment executes each process in each of the above embodiments based on a program stored in a storage medium, and a single device such as a personal computer or a plurality of devices is a network. Any configuration such as a connected system may be used.

  In addition, the computer in each embodiment includes an arithmetic processing device, a microcomputer, and the like included in the information processing device, and is a generic term for devices and devices that can realize the functions of the present invention by a program.

  In addition, although some embodiment of this invention was described, these embodiment is shown as an example and is not intending limiting the range of invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 1 ... Inspection inspection system, 2 ... Inspection inspection control system, 10 ... Information processing apparatus, 10a ... Processor, 10b ... Memory, 10c ... Storage device, 10d ... Input / output I / F, 10e ... Display device, 10f ... Input device, 10g ... wireless communication device, 10h ... communication device, 12 ... management device, 14 ... drone, 14a ... drone body, 14b ... rotor, 14c ... camera, 20 ... navigation management unit, 20a ... inspection inspection instruction input unit, 20b ... flight Plan processing unit, 21 ... acquired data organization management unit, 21a ... learning model storage unit, 21b ... detection target instruction input unit, 21c ... abnormal part detection unit, 21d ... acquired data storage unit, 21e ... image confirmation unit, 21f ... abnormal Location display unit, 21 g... Abnormal location storage unit, 21 h... Confirmation instruction input unit, 21 k... Confirmation data storage unit, 21 m. 22 ... patrol inspection management unit, 22a ... patrol inspection data storage unit, 22b ... patrol inspection result report creation unit, 30a ... control unit, 30b ... acquired data generation unit, 30c ... imaging control unit, 30d ... flight control unit, 30e ... Sensor group, 30f ... memory, 30g ... GPS data receiving unit, 30h ... wireless communication unit, 30k ... camera unit, 30m ... drive unit, 40a ... data storage unit, 40b ... abnormal location image data, 40c ... learning image data, 40d ... Teaching metadata, 40e ... Verification image data, 40f ... Learning image reading unit, 40g ... Teaching metadata reading unit, 40h ... Edge cutout unit, 40k ... Neighboring normal image search unit, 40m ... Normal image / abnormal image combination control 40n: Similar image generation unit, 40p: Learned model management unit, 40q: Verification image reading unit 40r ... inference execution unit, 40s ... index / conditions associated with the storage unit, 40t ... related correspondence unit, 40u ... inquiry unit, 40v ... learning model storage unit.

Claims (10)

A patrol inspection system for performing a patrol inspection of an inspection object based on image data taken by a camera mounted on an aircraft,
Including a management device and an information processing device,
The management device
Learning model generation means for generating a learning model for detecting an abnormality occurring in the inspection object from an image;
The information processing apparatus includes:
Storage means for storing a learning model generated by the management device;
Input means for inputting the image data;
An abnormal point detecting means for detecting image data of an abnormal point where an abnormality has occurred in the inspection object from the image data based on the learning model;
A patrol inspection system comprising: an abnormal part display means for displaying an image of the abnormal part based on the image data of the abnormal part. The information processing apparatus includes:
Setting means for setting flight control data for flying the aircraft in order to take an image including the inspection object;
2. The inspection inspection system according to claim 1, wherein the abnormal part detection means detects image data of the abnormal part based on the learning model corresponding to the inspection object photographed by flight based on the flight control data. . The information processing apparatus includes:
Further comprising an instruction input means for inputting an instruction indicating an abnormality type to be detected according to the inspection object;
The inspection system according to claim 1, wherein the abnormal part detection unit detects image data of an abnormal part corresponding to the abnormality type based on the learning model corresponding to the abnormality type.   The learning model generating means is based on a similar image that combines an abnormal image representing an abnormal location generated in the inspection object and at least one normal image that is not an abnormal location around the abnormal location. The inspection inspection system according to claim 1 which generates a learning model.   The learning model generating means combines the abnormal image and the normal image by a plurality of different ratios, and generates each of the abnormal types generated in the inspection object based on a similar image combined by any ratio. The inspection system according to claim 4, wherein a learning model is selected.   The abnormal location display means displays a confirmation screen that displays an image of an abnormal location candidate and an abnormal type based on the image data of the abnormal location detected by the abnormal location detection means, and a confirmation instruction for the confirmation screen The inspection inspection system according to claim 1, wherein: The abnormal part display means includes
Display a confirmation screen that displays the abnormal part candidate image and abnormality type based on the abnormal part image data detected by the abnormal part detection means, and input a confirmation instruction for the confirmation screen,
The inspection inspection system according to claim 4 or 5, wherein an image of the abnormal part candidate confirmed by the confirmation instruction is stored as an abnormal image used for generation of a similar image by the learning model generation unit.   The information processing apparatus inputs position data indicating the position of the aircraft at the time of photographing together with the image data from the aircraft, and corresponds to the image data of the abnormal part, the abnormality type, and the image data of the abnormal part. The inspection inspection system according to claim 3, further comprising inspection inspection result report generating means for generating inspection inspection result data including position data. An information processing apparatus for performing a patrol inspection on an inspection object based on image data taken by a camera mounted on an aircraft,
Storage means for storing a learning model for detecting an abnormality occurring in the inspection object from an image;
Input means for inputting the image data;
An abnormal point detecting means for detecting image data of an abnormal point where an abnormality has occurred in the inspection object from the image data based on the learning model;
An information processing apparatus comprising: an abnormal part display unit that displays image data of the abnormal part. Storage means for storing a learning model for detecting an abnormality occurring in the inspected object from the image in the computer;
Input means for inputting the image data;
An abnormal point detecting means for detecting image data of an abnormal point where an abnormality has occurred in the inspection object from the image data based on the learning model;
A patrol inspection control program for functioning as an abnormal part display means for displaying the image data of the abnormal part.