JP7440823B2 - Information processing device, information processing method and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

2. Description of the Related Art Conventionally, a technique is known in which an object to be inspected is inspected using a learning model generated using machine learning. When performing machine learning, learning data is prepared. For example, Patent Document 1 describes a technique for creating learning data consisting of image data of an object to be inspected and a label indicating whether the object to be inspected is a non-defective item or a defective item.

JP 2017-134508 Publication

Here, if there is an error (for example, an error in a label) in the created learning data, the performance of the learning model generated by machine learning will be worse than when there is no error in the learning data. For this reason, high-quality training data is required to generate a learning model.

Furthermore, in order to generate an inspection model for inspecting an inspection object by machine learning, it is necessary to prepare a large amount of learning data. At this time, it is extremely difficult for humans to collect high-quality learning data by visually searching for learning data (for example, images) that contains errors and correcting them, or by selecting learning data without errors. This is a complicated and time-consuming task. Therefore, it is required to efficiently collect high-quality learning data.

The technology described in Patent Document 1 is a technology intended to efficiently create learning data, but it is not a technology intended to efficiently collect high-quality learning data after the training data is created. .

Therefore, the present invention provides an information processing device, an information processing method, and a program that make it possible to efficiently collect high-quality learning data necessary for learning an inspection model used for inspecting an inspection object. The purpose is to

An information processing device according to one aspect of the present invention includes an image acquisition unit that acquires image data of an inspection target, and acquires inspection results of the inspection target obtained by inputting the image data into an inspection model of the inspection target. and a processing section that associates a test model with at least one of image data and test results, and stores the image data, test results, and test model in a storage section.

According to this aspect, since at least one of the image data and the inspection results used as learning data is associated with the inspection model, the user can collect desired learning data by tracing the inspection model. As a result, it becomes possible to efficiently collect high-quality learning data necessary for learning the inspection model used for inspecting the inspection object.

The above aspect may further include a learning section that performs a learning process to generate a new inspection model based on the image data, inspection results, and inspection model that the processing section has stored in the storage section.

According to this aspect, a test model can be efficiently generated by a learning process using high-quality learning data.

In the above aspect, the processing unit selects learning data for the learning process from the image data, inspection results, and inspection model stored in the storage unit based on information related to the inspection of the inspection target, and the selected learning data is The learning data acquisition unit may further include a learning data acquisition unit, and the learning unit may perform learning processing using the learning data.

According to this aspect, it becomes possible to obtain high-quality learning data more easily.

The above aspect further includes a correction unit that corrects the test results based on a user's operation to correct the test results, and the learning unit performs a learning process using learning data including the test results corrected by the correction unit. The processing unit then associates the corrected test results with the corresponding image data and the new test model, and stores the corrected test results, the corresponding image data, and the new test model. It may be stored in the section.

According to this aspect, by correcting the learning data, it becomes possible to collect learning data of better quality.

In the above aspect, the new test model generated by the learning unit may be shared with other information processing devices.

According to this aspect, the test model generated by the learning process is shared with other information processing devices.

In the above aspect, the processing unit acquires reliability-related data of at least one of the image data, the inspection result, and the inspection model, associates the acquired reliability-related data with corresponding data, and the learning unit The learning process may be performed based on at least one of image data, inspection results, and inspection models that are weighted using data related to the test.

According to this aspect, learning processing is performed based on weights depending on reliability, and a more appropriate test model is generated.

In the above aspect, the processing unit associates the plurality of inspection models and the inspection results corresponding to each of the plurality of inspection models with the image data used for the inspection using the plurality of inspection models, and performs the inspection using the plurality of inspection models. The storage unit may store the image data used in the test, the plurality of test models, and the test results corresponding to each of the plurality of test models.

According to this aspect, it becomes possible to check the inspection results for each of the plurality of inspection models for specific image data.

In the above aspect, the apparatus may further include a learning history generation section that generates a learning history of a new inspection model when a new inspection model is generated.

According to this aspect, it is possible to confirm under what conditions a new inspection model was generated.

An information processing method according to another aspect of the present invention acquires image data of the inspection object and obtains an inspection result of the inspection object obtained by inputting the image data into an inspection model of the inspection object. and associating the test model with at least one of the image data and the test results, and storing the image data, the test results, and the test model in a storage unit.

According to this aspect, since at least one of the image data and the inspection results used as learning data is associated with the inspection model, the user can collect desired learning data by tracing the inspection model. As a result, it becomes possible to efficiently collect high-quality learning data necessary for learning the inspection model used for inspecting the inspection object.

A program according to another aspect of the present invention causes a computer to acquire image data of the object to be inspected, and to obtain an inspection result of the object to be inspected obtained by inputting the image data into an inspection model of the object to be inspected. and associating the test model with at least one of the image data and the test result, and storing the image data, the test result, and the test model in the storage unit.

According to this aspect, since at least one of the image data and the inspection results used as learning data is associated with the inspection model, the user can collect desired learning data by tracing the inspection model. As a result, it becomes possible to efficiently collect high-quality learning data necessary for learning the inspection model used for inspecting the inspection object.

According to the present invention, an information processing device, an information processing method, and a program are provided that make it possible to efficiently collect high-quality learning data necessary for learning an inspection model used for inspection of an inspection object. can do.

FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to a first embodiment. 1 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of data associated with image data. FIG. 3 is a diagram showing changes in inspection models used in inspection lines placed in factories A and B. 2 is a diagram showing an example of data stored in a storage unit of cloud C. FIG. FIG. 1 is a diagram showing the physical configuration of an information processing device and a communication terminal according to the present embodiment. 3 is a flowchart showing an outline of processing according to the first embodiment. It is a flowchart which shows the 1st improvement process. This is an inspection history recording the inspection results using inspection models No. 4-1 and 5 regarding the image data with serial number 171030M_0001. This is historical data of the inspection model adopted on line B of factory B. It is a flowchart which shows the 2nd improvement process. FIG. 3 is a diagram illustrating an example of data stored in a storage unit of the cloud. FIG. 3 is a diagram illustrating an example of data in which a test model is associated with image data used for learning the test model. 7 is a flowchart showing processing of the information processing device according to the second embodiment.

Preferred embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in each figure, those with the same reference numerals have the same or similar configurations. In addition, when an alphabet is attached to a code, the alphabet is omitted unless there is a particular distinction. For example, when the information processing devices 20a and 20b are not distinguished, they are simply referred to as information processing device 20.

[First embodiment]
FIG. 1 is a diagram showing the configuration of an information processing system according to the first embodiment. In the first embodiment, it is assumed that products are produced in two factories A and B. Each factory is equipped with a plurality of inspection lines for inspecting manufactured products (also referred to as "inspection objects" or "workpieces"). The produced products are inspected on the inspection line based on the images. It is assumed that data related to testing, such as test results and image data, are aggregated in cloud C. Note that the location where test-related data such as test results and image data is stored is not limited to the cloud. Moreover, although two factories are shown in FIG. 1, the number of factories may be one or three or more.

FIG. 1 shows an imaging unit 10, an information processing device 20, a communication terminal 30, and a storage unit 40. In each of the two factories A and B, a plurality of information processing systems including an imaging section 10 and an information processing device 20 are arranged. In this embodiment, one information processing system is arranged for each inspection line. Note that a plurality of information processing systems may be arranged for one inspection line. Further, the number of information processing systems placed in each factory may be one.

Note that the imaging unit 10 and the information processing device 20 included in the information processing system may be an integrated device. Such integrated devices are also commonly referred to as smart cameras.

The information processing device 20 placed in each factory is communicably connected to the storage unit 40 of the cloud C via the network 50. Further, the communication terminal 30 used by the user is communicably connected to the information processing device 20 and the storage unit 40 via the network 50.

Network 50 can take various forms. The network 50 is, for example, a data transmission network (WAN) or a local area network (LAN) interconnected through dedicated lines. Here, the Internet, which is a representative public network, will be exemplified.

The imaging unit 10 includes various known cameras. The imaging unit 10 photographs the product and acquires image data in order to inspect the product being produced at the factory. The imaging unit 10 transmits the acquired image data to the information processing device 20.

The information processing device 20 performs an inspection using the image data of the inspection object, and transmits data such as the image data to the storage unit 40 of the cloud C. Further, the information processing device 20 can generate a new test model by acquiring necessary learning data from the storage unit 40 of the cloud C and performing learning processing. Furthermore, the information processing device 20 can also share the generated inspection model with other information processing devices 20.

The information processing device 20 includes an input section and a display section (not shown). The input unit accepts operations for inputting data from the user, and may include, for example, a mouse, a keyboard, and a touch panel. The input unit causes various functional units included in the information processing device 20 to perform various processes based on the received operation. The display unit visually outputs various information. For example, the display unit may be one that provides visual display, and specifically may be configured with an LCD (Liquid Crystal Display).

The functions of the information processing device 20 will be explained in more detail with reference to FIG. 2. As shown in FIG. 2, the information processing device 20 according to the present embodiment includes an image acquisition section 310, a result acquisition section 320, a processing section 330, a communication section 340, a search section 350, a learning data acquisition section 360, a correction section 370, It includes a learning section 380 and a storage section 390.

The image acquisition unit 310 acquires image data from the imaging unit 10 and transmits it to the result acquisition unit 320.

The result acquisition unit 320 acquires the inspection results of the inspection object obtained by inputting image data to the inspection model of the inspection object. Here, the inspection model is a model that inputs image data and outputs inspection results. The test model may be a learned model generated by a learning process, or may be a model configured with parameters set by a user, for example, without using a learning process. Specifically, the result acquisition unit 320 inputs image data to the inspection model and acquires the inspection results of the inspection object. The inspection result indicates, for example, whether or not there is an abnormality (for example, a defect) in the object to be inspected. Below, the case where there is no abnormality in the inspection object may be expressed as OK, and the case where there is an abnormality may be expressed as NG. The result acquisition unit 320 transmits the acquired test results to the processing unit 330 together with the test model and image data used in the test.

The processing unit 330 associates the test model with at least one of the image data and the test results. In this embodiment, a serial number is assigned to the product that is the object to be inspected. The processing unit 330 associates the product serial number with the image data. Furthermore, the processing unit 330 associates each image data with a corresponding test result and test model. In this embodiment, the user can set the processing unit 330 in advance so that the processing unit 330 associates various types of data by operating the input unit of the information processing device 20. Based on the settings, the processing unit 330 associates various data (data such as test results and test models) with the image data.

FIG. 3 is a diagram showing an example of data associated with image data. As shown in Figure 3, the image data includes the sensor type, sensor serial number, software version, name of the object to be inspected, number of the object to be inspected, name of the inspection factory, name of the inspection line, person in charge of inspection, and scene. Various data such as group name, scene name, test result, and reserve are associated. Here, "data reference" described in the data input method means that the processing unit 330 refers to predetermined data and associates it with image data. "Automatic data reference" is for the processing unit 330 to automatically refer to predetermined data and associate it with image data. Furthermore, "user's free input" is when the user operates the input unit of the information processing device 20 and associates information based on the operation with image data.

In addition to the data shown in Figure 3, the types of detected defects, the number of detected defects, the centroid coordinates of detected defects, shutter speed, gain, line takt, image processing time, timestamp, field of view, Data such as installation distance, lighting type, ambient temperature, ambient humidity, and free comments may be associated with the image data.

Returning to FIG. 2, each functional unit of the information processing device 20 will be explained. The processing unit 330 stores the image data, the test result, and the test model in the storage unit 40 of the cloud C via the communication unit 340 in a state where the test result and the test model are associated with the image data. Note that the image data, the test results, and the test model may be stored in the storage unit 40 of the cloud C in a state where they are not associated with each other.

Furthermore, the processing unit 330 acquires data related to the reliability of at least one of the image data, the test result, and the test model, and associates the obtained data related to reliability with the corresponding data. Specifically, the processing unit 330 acquires data related to the reliability of at least one of the image data, the test result, and the test model based on the input by the user to the input unit. Furthermore, the processing unit 330 associates data related to reliability with corresponding data, and transmits the data to the communication unit 340 as a definition file that defines reliability. The definition file transmitted to the communication unit 340 is transmitted to the storage unit 40 of the cloud C via the network 50.

The communication unit 340 transmits and receives various information. The communication unit 340 transmits, for example, the image data, test results, and test model associated by the processing unit 330 to the storage unit 40 of the cloud C via the network 50. Thereby, the image data, the test result, and the test model are stored in the storage unit 40 in a state where the image data, the test result, and the test model are associated with each other.

The search unit 350 searches for various information from the storage unit 40 of the cloud C based on the user's operation on the input unit. For example, the search unit 350 searches the storage unit 40 of the cloud C for data matching the search keyword based on the user's operation on the input unit for inputting the search keyword.

For example, when the user performs an operation to input a specific inspection model as a search keyword, the search unit 350 searches for image data associated with the specific inspection model, and acquires the searched image data. As described above, in this embodiment, the image data is associated with the inspection model. Therefore, the user can search for image data associated with the inspection model by searching using the inspection model as a search keyword.

The search unit 350 transmits the acquired image data to the learning data acquisition unit 360 as learning data as necessary. Note that the search unit 350 can use information related to various tests (hereinafter also referred to as "test information") such as a serial number, a date on which a search was performed, and a test line name as a search keyword.

The learning data acquisition section 360 selects learning data for learning processing from the image data, inspection results, and inspection model stored in the storage section 40 by the processing section 330 based on the inspection information of the inspection object. Get training data. In this embodiment, the learning data acquisition unit 360 acquires learning data such as image data and test results searched by the search unit 350. The learning data acquisition section 360 transmits the acquired learning data to the modification section 370 or the learning section 380.

The modification unit 370 modifies or deletes various data (image data, test results, etc.) included in the learning data based on the user's operation on the input unit. For example, the modification unit 370 modifies the test results based on a user's operation. In the present embodiment, the modification unit 370 modifies the test results that are included in the learning data acquired by the learning data acquisition unit 360 and are specified based on the user's operation. The modification unit 370 transmits learning data including the modified test results to the learning unit 380.

The learning section 380 performs a learning process to generate a new inspection model based on the image data, inspection results, and inspection model that the processing section 330 has stored in the storage section 40 . Specifically, the learning unit 380 generates a new inspection model by performing a neural network learning process using image data (input data) and inspection results (labels such as NG/OK) as learning data. . In this embodiment, learning processing is performed based on the learning data acquired by the learning data acquisition section 360 or the learning data modified by the modification section 370.

Further, the learning section 380 transmits the new test model generated by the learning process to the storage section 390 or the communication section 340. When the new inspection model is transmitted to the storage unit 390, the result acquisition unit 320 can inspect the object to be inspected using the new inspection model. Furthermore, when the new test model is transmitted to the communication unit 340, the communication unit 340 transmits the new test model to the other information processing devices 20, and the new test model is shared with the other information processing devices 20. .

Further, the learning unit 380 transmits the image data used for learning and the new inspection model to the processing unit 330. The processing unit 330 generates a learning history in which the new inspection model is associated with the image data used for learning. In this way, in this embodiment, the processing unit 330 has a function as a learning history generation unit. The processing unit 330 stores the generated learning history in the storage unit 40 of the cloud C via the communication unit 340.

Storage unit 390 stores various information. The storage unit 390 stores, for example, a test model used by the result acquisition unit 320 for testing and a new learning model generated by the learning unit 380. The various information stored in the storage unit 390 is referred to by the result acquisition unit 320 and the learning unit 380 as needed.

The functions of the information processing device 20 according to the present embodiment have been described above. Here, we will supplement the inspection model used in the above-mentioned factory. FIG. 4 is a diagram showing the changes in the inspection models used in the inspection lines located at factories A and B. Inspection lines arranged in factory A and factory B use an inspection model for inspecting products. This test model may be replaced with a new test model as necessary. FIG. 4 shows the numbers of inspection models used in each inspection line in 2016, 2017, and 2019.

For example, the No. 1 inspection model has been used for the old line A located in factory A from 2016 to 2019. In addition, the No. 2 inspection model has been used unchanged from 2017 to 2019 on the new line B, which was newly installed in 2017 at factory A.

On the other hand, the No. 3 inspection model was used in 2016 and 2017 for the old line A located in factory B, but in 2019, the inspection model of old line A was changed to the old line A of factory A. The test model used for line A is replaced with the same No. 1 test model.

At Factory B, a new line C will be added in 2017. Along with this expansion, the No. 4 inspection model used in the new line B was replaced with the No. 4-1 inspection model in 2017. At this time, the inspection model No. 4-1 is also used as the inspection model for the new line C. Thereafter, in 2019, the inspection models for new lines B and C were replaced with the No. 2 inspection model, which is the same as the inspection model used for new line B at factory A. It is assumed that various types of information such as image data, test results, and test models acquired in these test lines are collected in the storage unit 40 of cloud C.

FIG. 5 is a diagram showing an example of data stored in the storage unit 40 of cloud C. FIG. 5 shows, from left to right, the image data file name, inspection line name, inspection model, product serial number, inspection result, and time stamp. In this manner, in this embodiment, the test model is stored in association with the image data and test results. It is assumed that the inspection line names and inspection models shown in FIG. 5 correspond to the line names (A, B, or C) and inspection model numbers shown in FIG. 4. The inspection result is OK if the object to be inspected has no defects or the like, and NG if the object to be inspected has defects or the like. Furthermore, the timestamp represents the hour, minute, and second at the timing at which the test was performed, every two digits from the left. For example, 090100 represents 9:01:00. Note that the timestamp may include information representing the year, month, and day.

The communication terminal 30 performs various processes based on user operations. For example, the communication terminal 30 can transmit various types of information to the storage unit 40 of the cloud C via the network 50 based on a user's operation. Furthermore, the communication terminal 30 can also receive various types of information from the storage unit 40 of the cloud C via the network 50 based on a user's operation.

FIG. 6 is a diagram showing the physical configurations of the information processing device 20 and the communication terminal 30 according to this embodiment. The information processing device 20 and the communication terminal 30 include a CPU (Central Processing Unit) 90 corresponding to a calculation section, a RAM (Random Access Memory) 91 corresponding to a storage section, and a ROM (Read only Memory) 92 corresponding to a storage section. , a communication section 93 , an input section 94 , and a display section 95 . These components are connected to each other via a bus so that they can transmit and receive data. Note that in this embodiment, a case will be described in which the information processing device 20 is composed of one computer, but the information processing device 20 may be realized by combining a plurality of computers. Further, the configuration shown in FIG. 6 is an example, and the information processing device 20 may have configurations other than these, or may not have some of these configurations.

The CPU 90 is a control unit that controls the execution of programs stored in the RAM 91 or ROM 92, and performs calculations and processing of data. The CPU 90 executes programs for realizing the processing of each functional unit described with reference to FIG. Specifically, the CPU 90 executes a program for inspecting an object (image inspection program) based on image data, executes a program for associating various data, and executes an inspection through learning processing based on learning data. Execute a program that generates a model (model generation program). Further, the CPU 90 receives various data from the input section 94 and the communication section 93, and displays the calculation results of the data on the display section 95 or stores them in the RAM 91.

The RAM 91 is a storage unit in which data can be rewritten, and may be formed of, for example, a semiconductor storage element. The RAM 91 may store data such as programs executed by the CPU 90 and images of objects. Note that these are just examples, and the RAM 91 may store data other than these, or some of them may not be stored.

The ROM 92 is a storage section from which data can be read, and may be formed of, for example, a semiconductor storage element. The ROM 92 may store, for example, an image inspection program, a model generation program, and data that will not be rewritten.

The communication unit 93 is an interface that connects the information processing device 20 or the communication terminal 30 to other devices. The communication unit 93 may be connected to a communication network such as the Internet.

The image inspection program and the model generation program may be provided while being stored in a computer-readable storage medium such as the RAM 91 or ROM 92, or may be provided via a communication network connected by the communication unit 93. In the information processing device 20, the various operations described using FIG. 2 are realized by the CPU 90 executing programs. Specifically, the CPU 90 realizes the operations of the image acquisition unit 310, result acquisition unit 320, processing unit 330, search unit 350, learning data acquisition unit 360, correction unit 370, and learning unit 380 shown in FIG. Note that these physical configurations are merely examples, and do not necessarily have to be independent configurations. For example, the information processing device 20 may include an LSI (Large-Scale Integration) in which a CPU 90, a RAM 91, and a ROM 92 are integrated.

Note that the storage unit 40 of the cloud C is composed of various known storage media.

Next, processing using the information processing system according to this embodiment will be described with reference to the flowchart in FIG. Various situations are assumed for the situation in which the process shown in FIG. 7 is started. For example, there may be a situation where a product produced in a factory has a defect in the market. Further, even though a specific product is determined to be NG in a factory inspection, there are cases in which the product is determined to be OK in a re-inspection. Furthermore, there may be a situation in which a specific product is determined to be OK in a factory inspection, but the product is determined to be NG in a re-inspection. Here, we will explain a case where a product produced in a factory causes a problem in the market.

Here, it is assumed that the product corresponding to the serial number 171030M_0001 surrounded by a square shown in FIG. 5 has a problem in the market. As shown in FIG. 5, the test result for serial number 171030M_0001 using test model No. 4-1 is OK. However, in reality, the product with serial number 171030M_0001 is a defective product, and the correct test result is NG. Therefore, the test model is performing an incorrect test.

In the following, it is assumed that people involved in production or inspection of products in a factory (line managers, quality managers, inspectors, and engineers) perform various processes using the information processing device 20 or the communication terminal 30 they own. explain.

First, a factory quality manager identifies the serial number of a defective product (step S101). Here, the quality manager specifies serial number 171030M_0001.

Next, the quality manager identifies the image data of the defective product (step S103). Specifically, the quality manager uses the search software of the communication terminal 30, for example, to search for the identified serial number 171030M_0001, and identifies the image data of the defective product from the storage unit 40 of the cloud C.

Next, it is determined whether or not there is an abnormality in the identified image data (step S105). For example, an inspector may visually determine an abnormality in the image data. Furthermore, the inspector may perform a reexamination using the specified image data.

If it is determined that there is no abnormality in the identified image data (step S105: NO), a first improvement process is performed (step S107). On the other hand, if it is determined that there is an abnormality in the image data (step S105: YES), a second improvement process is performed (step S109).

When the first improvement process or the second improvement process ends, the process shown in FIG. 7 ends.

The first improvement process will be described with reference to FIG. 8. If the first improvement process is performed, it is determined in step S105 that there is no abnormality in the identified image data. Therefore, it is considered that there is an abnormality in the test model that performed the incorrect test. In the first improvement process, a new test model is generated by a learning process, and a process of replacing this inappropriate test model with a new test model is performed.

First, an inspector identifies an inappropriate inspection model (step S201). Specifically, based on the image data identified in step S103, the storage unit 40 searches for inspection models associated with the image data using the search software of the communication terminal 30, and searches for inappropriate inspection models. Specify as. Here, it is assumed that test model No. 4-1 is identified as an inappropriate test model.

Next, the inspector selects the image data of the inspection object inspected by the inappropriate inspection model (step S203). Here, image data with serial numbers (171030M_0001, 171030M_0524, 191029M_0001, 191029M_0494) inspected by inspection model No. 4-1 is selected.

Next, the inspector re-inspects the inspection object that was inspected by the inappropriate inspection model (step S205). Specifically, the inspector visually re-inspects the image data of the object to be inspected. As a result, it is assumed that serial numbers 171030M_0001 and 171030M_0524 are determined to be NG.

Next, the inspector confirms that there is no successor test model to the inappropriate test model (step S207). If there is a successor test model (step S207: NO), the process advances to step S209. On the other hand, if there is no successor test model (step S207: YES), the process advances to step S213. The following will mainly describe the case where it is confirmed that there is no successor test model, but first the processing (steps S209 and S211) when it is confirmed that there is a successor test model will be described.

When it is confirmed in step S207 that there is a successor test model, the inspector confirms the certainty of the successor test model (step S209). In step S207, for example, if it is confirmed that there is an inspection model No. 2 as a successor to the inspection model No. 4-1, the inspector uses the inspection model No. 2 to inspect the defective product (serial number 171030M_0001, 171030M_0524). For example, the inspector inputs the image data of the defective product and the No. 2 inspection model into the processing unit 330, and confirms the reliability of the No. 2 inspection model by having the processing unit 330 perform a re-inspection.

Next, the inspector determines whether the successor inspection model is appropriate (step S211). For example, when the No. 2 inspection model determines that all defective products are NG, the No. 2 inspection model is determined to be appropriate. On the other hand, if the No. 2 inspection model determines that any defective product is OK, the inspector determines that the No. 2 inspection model is inappropriate. If it is determined that the successor test model is appropriate (step S211: YES), the process advances to step S223. On the other hand, if the successor test model is determined to be inappropriate (step S211: NO), the process returns to step 203.

Next, a description will be given of processing when a successor test model is not confirmed in step S207. In this case, the inspector causes the information processing device 20 to acquire learning data (step S213). Specifically, the search unit 350 of the information processing device 20 searches the image data, inspection results, and inspection model stored in the storage unit 40 of the cloud C by the processing unit 330 based on the inspection information input by the inspector. Select learning data for learning processing and acquire the selected learning data. For example, the search unit 350 acquires learning data from the storage unit 40 using inspection information (eg, inspection model, time of inspection, inspection line, etc.) input based on an operation by an inspector as a search keyword. Since the inspection information is associated with the image data and inspection results that serve as learning data, the learning data acquisition unit 360 can acquire the learning data using the search information. The learning data acquired by the search unit 350 is transmitted to the learning data acquisition unit 360.

Next, the inspector causes the information processing device 20 to modify the inspection results (step S215). Specifically, the inspector causes the correction unit 370 of the information processing device 20 to correct the incorrect test results included in the learning data by performing an operation to correct the test results. For example, the correction unit 370 corrects an erroneous test result to a result determined by the inspector based on an operation by the inspector.

Next, the engineer causes the information processing device 20 to generate a new inspection model through learning processing (step S217). Specifically, the learning unit 380 of the information processing device 20 performs a learning process using the learning data including the test results corrected in step S213, and generates a new test model as the No. 5 test model. At this time, the learning unit 380 generates a learning history of a new test model. The generated learning history is transmitted via the communication unit 340 and stored in the storage unit 40 of the cloud C.

Next, the engineer causes the information processing device 20 to re-examine the image data that was incorrectly inspected, based on the new inspection model generated by the learning unit 380 (step S219). Here, the new inspection model is transmitted to the result acquisition unit 320, and the result acquisition unit 320 uses the new inspection model to reexamine the image data on which the incorrect inspection was performed.

Next, the engineer determines whether the new inspection model is appropriate based on the results of the re-inspection (step S221). For example, if the retest result is incorrect, it is determined that the new test model is not appropriate, and the process returns to step S213. On the other hand, if the retest result is correct, it is determined that the new test model is appropriate, and the process advances to step S223.

Next, the line manager determines the suitability of the inspection model for each inspection line (step S223). For example, if it is confirmed in step S221 that the new inspection model is appropriate, the line manager determines whether to apply the new inspection model. Furthermore, if an appropriate successor inspection model has been confirmed in step S211, the line manager determines whether the successor inspection model is to be applied.

Next, the line manager applies the new inspection model or the successor inspection model to the information processing device 20 of each inspection line (step S225). More specifically, the line manager causes the processing unit 330 of the information processing device 20 corresponding to each inspection line to acquire the inspection model determined to be applicable in step S223. Here, when the new test model is applied to another information processing device 20 different from the information processing device 20 that generated the new test model, the information processing device 20 that generated the new test model 340, the new test model is transmitted to other information processing devices 20. As a result, the new inspection model is applied (shared) to other information processing apparatuses 20.

Next, the line manager updates various data stored in the storage unit 40 (step S227). For example, the line manager updates various data stored in the cloud C using the information processing device 20 or the communication terminal 30. FIGS. 9 and 10 show examples of updated data.

FIG. 9 is an inspection history in which inspection results using inspection models No. 4-1 and 5 are recorded for image data with serial number 171030M_0001. In the inspection history, the inspection results based on the new inspection model No. 5 generated by the learning process are newly recorded in addition to the inspection results based on the inspection model No. 4. Although FIG. 9 shows test results using two test models, the test history may also record test results using three or more test models.

FIG. 10 shows historical data of the inspection model adopted in line B of factory B. As shown in FIG. 10, the test model numbers change from 4 to 4, 4-1, and 5 in order from the oldest model.

The first improvement process (step S107) shown in FIG. 8 has been described above. Next, with reference to FIG. 10, the second improvement process (step S109) shown in FIG. 11 will be described.

FIG. 12 is a diagram showing data stored in the storage unit 40 of cloud C when performing the processing in FIG. 11. Here, it is assumed that the image data corresponding to the serial number 171030M_0001 surrounded by a solid square is abnormal image data, and this image data is determined to be abnormal in step S105.

First, the quality manager identifies the inspection line and time stamp from which abnormal image data was acquired (step S301). Specifically, the quality manager uses, for example, the communication terminal 30 to search for the inspection lines and inspection lines stored in association with the serial number in the storage unit 40 using the specified serial number of the image data as a search keyword. Identify timestamps.

Next, the quality manager obtains image data obtained at the specified inspection line before and after the specified time stamp (step S303). Specifically, the quality manager causes the communication terminal 30 to acquire image data acquired at a time stamp near the specified time stamp for the specified inspection line (line B).

Next, the inspector determines whether there is any abnormality in the image data acquired in step S305 (step S305). For example, the inspector visually checks whether there are any abnormalities in the acquired image data. If it is determined that there is no abnormality in the image data (step S305: NO), the process returns to step S303. If it is determined that there is no abnormality in the image data (step S305: YES), the process advances to step S307.

If the determination is YES in step S305, abnormal image data is identified (step S307). Here, it is assumed that only the image data with serial number 171030M_0001 is identified as abnormal image data.

Next, the inspector identifies the inspection model generated by the learning process using the abnormal image data (step S309). Specifically, the inspector uses the serial number of the image data with the identified abnormality to identify the inspection model generated by the learning process using the image data with the abnormality from the storage unit 40. At this time, the inspector searches for and specifies a desired inspection model in the communication terminal 30 based on data that associates the inspection model stored in the storage unit 40 with the image data used for learning the inspection model. let

FIG. 13 is a diagram showing an example of data in which a test model is associated with image data used for learning the test model. In the example shown in FIG. 13, the file name of image data associated with inspection model No. 4-1 is shown. FIG. 13 shows that a total of 494 pieces of image data with serial numbers 161029M_0001 to 161029M_0494 are associated with the inspection model. The communication terminal 30 refers to such data and searches for an inspection model whose image data used for learning includes image data with serial number 171030M_0001.

Next, the process of step S311 is performed, but since the process of step S311 is substantially the same as the process of step S213 described with reference to FIG. 8, the description will be omitted here.

Next, the inspector causes the information processing device 20 to modify the learning data (step S313). For example, the inspector causes the correction unit 370 of the information processing device 20 to correct incorrect test results included in the learning data by performing an operation to correct the test results. Specifically, the correction unit 370 corrects the incorrect test result to the result determined by the inspector based on the operation by the inspector. The inspector can also modify the learning data so as to delete from the learning data image data and inspection results that are inconvenient during learning.

Next, the processing from steps S315 to S325 is performed, but since the processing from steps S315 to S325 is substantially included in the processing from steps S217 to S227 described with reference to FIG. 8, the description thereof will be omitted here. Note that in the processing from steps S321 to S325 shown in FIG. 11, the processing related to the successor test model in steps S223 to S227 described with reference to FIG. 8 is not performed.

The processing of the information processing system according to this embodiment has been described above.

Conventionally, although there are problems such as storage capacity, there is an increasing demand for logging all data such as image data or test results. The cloud is a promising candidate for storing logged data. Here, it is assumed that the logging results of each production base will be saved in the cloud. However, in general, the file naming method for saved data or the linking between saved data and inspection-related information is not unified for each production site, inspection line, and sensor employed.

When responding to product defects that occur in the market, it is necessary to retrieve desired data from data stored in the cloud. When selecting and obtaining the necessary data from the cloud, if the recording method is not unified, the necessary data will not be obtained unless the person in charge at each production site takes time to connect, which can be extremely time consuming. It may take some time. As a result, there is a risk that the company will incur large losses, such as an increase in the number of items to be recalled (defective products) due to delays in initial response, delays in restarting production due to delays in taking countermeasures, or additional outflow of similar defective products at other production bases. .

According to this embodiment, the processing unit 330 associates the test model with at least one of the image data and the test result, and causes the storage unit 40 to store the image data, the test result, and the test model. This allows the user to follow the test model and collect desired learning data. As a result, it becomes possible to efficiently collect high-quality learning data necessary for learning the inspection model used for inspecting the inspection object. Conventionally, it was necessary for a person in charge to analyze a huge amount of past records, but according to this embodiment, the number of man-hours required to acquire learning data is reduced, and as a result, the period required to acquire learning data is shortened. be shortened. As a result, various risks are reduced.

According to this embodiment, since the learning data is acquired through processing by the information processing device 20, omissions due to human error can be prevented, and the risk of human error can be reduced. Furthermore, according to this embodiment, the inspection model is smoothly generated by machine learning, so that the quality of inspection can be improved at an early stage.

Furthermore, in order to generate a test model through learning processing, it is necessary to associate image data with labels such as normal/abnormal. When performing learning processing using huge amounts of data collected at manufacturing sites, a huge amount of man-hours are required to extract data that is effective for learning and to process data into a form that can be used for machine learning. . For example, when a product is inspected using an image sensor, it is necessary to label images taken during the product inspection as normal or abnormal. Furthermore, for products with abnormalities, it is necessary to indicate where in the image the abnormality is. However, if it is unclear whether the stored image data of the inspection target is a normal product or an abnormal product, it is necessary to separately confirm the image.

In this embodiment, since the image data and the test results are stored in the storage unit 40 in an associated state, the user can obtain the normal/abnormal test results together with the image data. As a result, the effort of separately checking the image is reduced.

Furthermore, conventionally, when a product with a specific serial number is found to be defective on the market, it is necessary to investigate which inspection line of the factory the inspection model was used for inspecting the product. However, from image data alone, the inspection line and inspection model on which the defective product was inspected may not be known unless the person in charge of the factory that produced the defective product investigates. In this embodiment, since the inspection model is associated with the image data stored in the cloud, a problematic inspection model can be quickly traced.

Furthermore, in order to investigate the influence range of a defective product, it may be necessary to specify image data of an inspection object inspected with a specific inspection model. In this embodiment, the inspection model used in each inspection line and the image data are stored in the storage unit 40 of the cloud C in association with each other, so image data inspected with a specific inspection model is searched and extracted. be able to. Therefore, the learning image data necessary for updating the inspection model can be quickly selected from the huge amount of data stored in the cloud.

[Second embodiment]
In the second embodiment, a description of common matters with the first embodiment will be omitted, and only different points will be explained.

Processing in the second embodiment will be described with reference to the flowchart in FIG. 14. The process shown in FIG. 14 is started, for example, when a new inspection line is launched in a factory.

First, the production manager defines the reliability of various items using a definition file (step S401). More specifically, the production manager uses the information processing device 20 or the communication terminal 30 to store in the storage unit 40 a definition file that defines the reliability of various items stored in the storage unit 40 of the cloud C. let The reliability of various items includes, for example, the reliability of image data, the reliability of inspection models, the reliability of products from the inspection line, and the reliability of inspection results.

The contents of the definition file for various items may be, for example, a correspondence between reliability expressed as a percentage and the date when the reliability was updated. Further, the names of definition files for various items may be information regarding the various items. For example, the name of the definition file that defines the reliability of image data may include a name such as "171030M_0001_Image_Reliability". Further, the name of the definition file of the inspection model may include, for example, "No. 4-1 inspection model", and the name of the definition file that defines the reliability of the product produced on the production line may include " "Line B of Factory A" may be included. In this way, various items and reliability are associated and stored in the storage unit 40 of cloud C.

After the definition file is stored in the storage unit 40, when the definition file update time arrives, the process proceeds to step S403, and the production manager uses the definition file to update the reliability of various items (step S403). Specifically, the production manager uses the communication terminal 30 to store a new definition file in which reliability has been updated for various items in the storage unit 40 of the cloud C.

Next, the engineer causes the information processing device 20 to acquire learning data and reliability (step S405). Specifically, the search unit 350 acquires learning data and reliability from the storage unit 40 based on the operation by the engineer, and transmits the learning data and reliability to the learning unit 380.

The engineer causes the learning unit 380 to generate a new inspection model through learning processing based on the reliability of various items and learning data (step S407). More specifically, the learning unit 380 performs learning processing using data weighted using reliability data. For example, when the learning unit 380 uses test results of a test model as learning data, the test results are weighted according to the reliability of the test model.

For example, the learning unit 380 can binarize the reliability of the learning data based on a threshold value. Specifically, regarding the reliability of various learning data, the learning unit 380 determines that reliability exceeding a threshold value is reliable, and that reliability that is equal to or less than the threshold value is unreliable. The learning unit 380 can perform learning processing using only reliable learning data.

Furthermore, when generating a test model by modifying the weighting parameters of the neural network using the learning data using the error backpropagation method, the learning unit 380 adjusts the amount of modification of the weighting parameters according to the reliability of the data. You can also adjust it. Specifically, the learning unit 380 may increase the amount of modification of the weighting parameter as the reliability of the learning data is higher, and decrease the amount of modification of the weighting parameter as the reliability of the learning data becomes lower.

Furthermore, in learning a neural network, there is a method of repeatedly inputting the same learning data to the neural network and modifying the weights. When this method is used, the learning unit 380 may adjust the number of iterations depending on the reliability of the data. Specifically, the learning unit 380 may increase the number of repetitions as the reliability of the learning data is higher, and decrease the number of repetitions as the reliability of the learning data becomes lower.

Alternatively, the learning unit 380 can perform the learning process by setting the test results used for learning as values between the two values instead of the binary values of OK/NG, depending on the reliability of various items. The learning unit 380 performs a learning process using the learning data weighted in this manner and generates a new test model. This produces a more appropriate test model than if confidence were not used.

Next, the processing from steps S409 to S413 is performed, but since the processing from steps S409 to S415 is substantially the same as the processing from steps S319 to S325 shown in the flowchart of FIG. 11, the description thereof will be omitted here.

Next, the line manager updates various data using the communication terminal 30 (step S415). Specifically, the line manager uses the communication terminal 30 to update various types of information associated with the image data stored in the storage unit 40 of the cloud C and definition files of various items. At this time, the image data may be backed up in the storage unit 40 of the cloud C.

The embodiments described above are intended to facilitate understanding of the present invention, and are not intended to be interpreted as limiting the present invention. Each element included in the embodiment, its arrangement, material, conditions, shape, size, etc. are not limited to those illustrated, and can be changed as appropriate. Further, it is possible to partially replace or combine the structures shown in different embodiments.

In the embodiment described above, the processing by the various functional units shown in FIG. 2 has been described as being performed by one information processing device 20. The present invention is not limited to this, and a part of the processing by the various functional units shown in FIG. 2 may be performed by other devices. For example, the processing by the learning data acquisition unit 360, the correction unit 370, and the learning unit 380 may be realized by an information processing device different from the information processing device that realizes these other functional units. Further, in the above embodiment, the definition file is created by the processing unit 330 included in the information processing device 20, but the definition file may be created by another device (for example, the communication terminal 30). Further, in the above embodiment, the search unit 350 searches for and acquires learning data, and the acquired learning data is transmitted to the learning data acquisition unit 360. However, the learning data acquisition unit 360 searches for learning data ( selection).

Further, in the above embodiment, the explanation has been given mainly on the assumption that the test result of the test model is OK or NG. If the test result is NG, the mode (type) of the abnormality occurring in the test object may be determined. In this case, the test results may include the type of abnormality. The types of abnormalities include, for example, scratches, dirt, and dents. It is also possible to apply to the information processing device 20 according to the above embodiment an inspection model that can identify, for example, the type of abnormality by solving a classification problem (multi-class classification problem).

Furthermore, in the embodiment described above, it is assumed that one inspection model corresponds to one inspection line. The present invention is not limited to this, and a plurality of inspection models may be associated with one inspection line. For example, in one inspection line, multiple types of products (inspection objects) may flow, and each type of product may be inspected by the information processing device 20. In this case, one test model may be associated with one product type. The information processing device 20 can perform inspection using inspection models corresponding to each product type.

Further, a plurality of inspection models may be associated with one product type. For example, when inspecting the top surface and side surface of an object to be inspected of one type, an imaging unit (for example, a total of two cameras, a first camera and a second camera) may be associated with each surface. For example, the first inspection model is associated with the first camera, and the second inspection model is associated with the second camera. In this case, the information processing device 20 inspects the top surface of the inspection target using the image captured by the first camera and the first inspection model, and the side surface of the inspection target using the image captured by the second camera and the second inspection model. Can be inspected. Note that the number of cameras used in the inspection line may be three or more, and each camera may be associated with an inspection model.

Furthermore, a plurality of inspection models may be associated with one camera. For example, an inspection model may be associated with each region of the inspection object imaged by the camera. For example, when a camera images two areas (a first area and a second area), different inspection models may be associated with each of the first area and the second area. Note that the number of regions imaged by the camera may be three or more, and each region may be associated with an inspection model.

In this way, the technique of the present invention can be used even when a plurality of inspection models correspond to each inspection line. In this case, in addition to the data shown in FIGS. 5 and 12, the storage unit 40 of the cloud C may store various information associated with the image data. For example, the type of object to be inspected, information specifying the camera, the area imaged by the camera, and the like may be stored in the storage unit 40 in a state where they are associated with respective inspection models. Thereby, image data associated with a desired inspection model can be efficiently collected as learning data.

For example, assume that factory A has inspection line D and factory B has inspection line E. Furthermore, it is assumed that inspection models K1 and L are used in inspection line D of factory A, and inspection models K2 and M are used in inspection line E of factory B. At this time, it is assumed that the test models K1 and K2 are used to perform similar tests. In this case, new inspection models can be generated by a learning process using image data acquired by the imaging units of inspection lines D and E, and used as inspection models K1 and K2.

[Appendix]
an image acquisition unit (310) that acquires image data of the inspection object;
a result acquisition unit (320) that acquires an inspection result of the inspection object obtained by inputting the image data into an inspection model of the inspection object;
a processing unit (330) that associates the test model with at least one of the image data and the test results, and stores the image data, the test results, and the test model in a storage unit;
An information processing device (20) comprising:

DESCRIPTION OF SYMBOLS 10... Imaging unit, 20... Information processing device, 30... Communication terminal, 40... Storage unit, 50... Network, 310... Image acquisition unit, 320... Result acquisition unit, 330... Processing unit, 340... Communication unit, 350... Search section, 360...learning data acquisition section, 370...correction section, 380...learning section, 390...storage section

Claims (9)

an image acquisition unit that acquires image data of the inspection target;
a result acquisition unit that acquires an inspection result of the inspection object obtained by inputting the image data into an inspection model of the inspection object;
a processing unit that associates the test model with at least one of the image data and the test results, and stores the image data, the test results, and the test model in a storage unit;
Equipped with
The processing unit associates the plurality of inspection models and the inspection results corresponding to each of the plurality of inspection models with the image data used for the inspection using the plurality of inspection models, and performs the inspection using the plurality of inspection models. The information processing device stores image data used in the storage, the plurality of inspection models, and inspection results corresponding to each of the plurality of inspection models in the storage unit . a learning unit that performs a learning process to generate a new inspection model based on the image data, the inspection results, and the inspection model that the processing unit has stored in the storage unit;
The information processing device according to claim 1, further comprising: Based on information related to the inspection of the inspection object, the processing unit selects learning data for the learning process from the image data, the inspection results, and the inspection model stored in the storage unit. further comprising a learning data acquisition unit that acquires learning data,
The learning unit performs the learning process using the learning data.
The information processing device according to claim 2. further comprising a correction unit that corrects the test result based on a user's operation to correct the test result,
The learning unit generates a new inspection model by performing the learning process using learning data including the inspection results corrected by the correction unit,
The processing unit associates the corrected test results with the corresponding image data and the new test model, and stores the corrected test results, the corresponding image data, and the new test model in the storage unit. let,
The information processing device according to claim 3. The new inspection model generated by the learning unit is shared with other information processing devices;
The information processing device according to any one of claims 2 to 4. The processing unit acquires reliability-related data of at least one of the image data, the test results, and the test model, and associates the obtained reliability-related data with corresponding data;
The learning unit performs the learning process based on at least one of the image data, the inspection result, and the inspection model weighted using the reliability-related data.
The information processing device according to any one of claims 2 to 5. further comprising a learning history generation unit that generates a learning history of the new inspection model when a new inspection model is generated;
The information processing device according to any one of claims 2 to 6 . Obtaining image data of the object to be inspected;
Obtaining an inspection result of the inspection object obtained by inputting the image data into an inspection model of the inspection object;
associating the test model with at least one of the image data and the test results, and storing the image data, the test results, and the test model in a storage unit;
including;
The storing includes associating the plurality of inspection models and the inspection results corresponding to each of the plurality of inspection models with the image data used in the inspection using the plurality of inspection models, and storing the image data using the plurality of inspection models. An information processing method comprising storing image data used in an inspection, the plurality of inspection models, and inspection results corresponding to each of the plurality of inspection models in the storage unit . to the computer,
Obtaining image data of the object to be inspected;
Obtaining an inspection result of the inspection object obtained by inputting the image data into an inspection model of the inspection object;
associating the test model with at least one of the image data and the test results, and storing the image data, the test results, and the test model in a storage unit;
A program for executing
The storing includes associating the plurality of inspection models and the inspection results corresponding to each of the plurality of inspection models with the image data used in the inspection using the plurality of inspection models, and storing the image data using the plurality of inspection models. A program comprising: storing image data used in an inspection, the plurality of inspection models, and inspection results corresponding to each of the plurality of inspection models in the storage unit .