RU2823907C1 - Computer-implemented method of automatic control of assembly production - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: invention relates to computer engineering for quality control of assembly products. Technical result is achieved by the fact that the disclosed solution provides for the following stages, on which images from video cameras are captured, parts of the assembled product are identified on images from cameras based on pre-trained models of machine vision; filtering the recognized data based on a sliding window, the size of which is selected automatically based on the speed of pattern recognition; determining the correctness of the performed operation by detecting the intermediate result of the assembly of the product; evaluating the assembly of the product and the completeness of intermediate operations; Signal of successful assembly of part is generated or signal of erroneous operation is generated.

EFFECT: higher quality of assembly by monitoring the correct execution of the final set of regulated assembly operations.

4 cl, 6 dwg

Description

The invention relates to the field of computer technology for quality control of assembly products.

Currently, there are several different approaches to building automation systems for monitoring production assembly operations. Most of them use artificial intelligence technologies, augmented, mixed and virtual reality and function as decision support systems.

The OCAR Cable system is known, which can provide the ability to identify cables and wires in bundles using augmented reality technologies. The disadvantage of the solution is that it cannot be used for other products and devices.

Intelligent methods for detecting defects and process failures are used in the food industry, for example, when detecting substandard products in a tray with eggs (patent RU 2759733). The method consists in the fact that filled egg trays on a conveyor pass through a detection system containing a video camera connected to a machine vision controller that processes the image obtained from the video camera and detects markings on the eggs, facts of under-insertion in the tray, and pieces of dirt. The food industry can also use a solution based on patent RU 2775808 "Neural network systems and computer-implemented methods for identifying and/or assessing one or more food products in visual input information." The method is designed to process a video stream for identifying and assessing the quality of pizza production. The disadvantage of both methods is their focus on a specific set of operations and technical requirements, a strictly limited and unchangeable set of operations. In addition, both solutions use a different technical method of detecting defects from the proposed one. The first method uses a deterministic detection algorithm based on geometric features of objects. The second method uses one or more sequential detectors based on a neural network of the CNN class.

The "Software package for monitoring manual operations based on computer vision and INS technologies" is known (registration certificate No. 2021614855). It is designed to provide visual support for manual operations in the assembly/disassembly processes of complex technical products and components of aviation equipment (including components of on-board radio-electronic equipment, avionics), processes of maintenance and repair of these products and components based on computer vision technologies and artificial neural networks.

The disadvantage of this solution is that it cannot be used for other technical products except those on which the computer vision system was trained. Unlike the proposed method, it is not universal, but highly specialized.

The technical task to which the claimed invention is directed is to automate the control of assembly production of any type without reference to the features of the product.

The technical result consists in increasing the efficiency of assembly by detecting errors at different stages.

The technical result is achieved through a method of automatic control of assembly production (see Fig. 1), according to which:

- the module (10) of the database of template sequences of operations for successful assembly of parts is loaded,

- images are captured from video cameras using the module (2),

- the operator selects the type of product to be assembled using module (2) and the corresponding pre-trained machine vision model,

- the operator places the product parts on the table in the working area of the video camera capture according to the assembly instructions;

- recognition of parts of the assembled product is carried out using a module (3) in images from cameras based on pre-trained machine vision models;

- using module (4), filtering of recognized data is performed based on a sliding window, the size of which is selected automatically based on the speed of pattern recognition, but not less than 4 frames;

- using module (5), the intermediate result of the product assembly is detected in order to determine the correctness of the operation performed;

- with the help of module (6) the accumulation of data on intermediate operations occurs;

- with the help of module (7), the assembly of the product and the completion of intermediate operations are assessed, while the accumulated data in module (6) and the data from module (1) about the assembled part are checked;

- a signal of successful assembly of the part is generated if the sequentially accumulated data coincides with the template operations for assembling the part;

- a signal about an erroneous operation is generated if the chain of operations in module (6) differs from the template.

The method of automatic control of assembly production additionally includes visualization of the current state of the product assembly using module (8) in graphical form.

The method of automatic control of assembly production additionally includes recording the current state of the product assembly in the event log using module (9).

The method of automatic control of assembly production additionally includes sending assembly data to external systems using module (11).

The method is implemented using a system (see Fig. 1) consisting of a module (1) for selecting an assembly operation with an interface for the user, wherein the selection of the manufactured part is made until the cycle is interrupted by the operator. Module (2) for capturing images from cameras on a stand installed above the working surface of the assembly zone, wherein the capture of images from 4k cameras occurs above the assembly zone. Module (3) for recognizing patterns in images from cameras with the output of data on localization and classification of objects in images, wherein the angle of installation of cameras is selected based on the task. The default angle is 90° (perpendicular to the level of the working surface of the assembly zone), but can be changed in some versions based on technical requirements. Module (4) for filtering recognized data based on a sliding time buffer (sliding window), wherein the buffer size is selected automatically based on the speed of pattern recognition, but not less than 4 frames. Patterns are filtered based on intersection data. A recognized object is considered to exist if the object has an intersection with a similar class of object in the buffer for more than 50% of frames, or is considered to be falsely detected if it was absent in 50% of frames or more. Assembly operation detection module (5), which generates an event of a correct assembly stage based on the detected images. The output of this module is events about the current operation on the stand. Module (6) is a memory buffer with information about previously accumulated intermediate operations on the stand. Module (7) for assessing the state (status) of the product assembly and the completeness of intermediate operations, which checks the accumulated data in module (6) and the data from module (1) about the part being assembled, generates a signal of successful part assembly if the sequentially accumulated data coincide with the template operations for part assembly; generates a signal about an erroneous operation if the chain of operations in module (6) differs from the template. The output signal from this module generates an instruction for the visualization module to display data on the stand. The output signal also sends data to the system for accumulating monitoring data about the production line. Module (8) is a visualization and prompting system for the stand operator. This module uses a projector to display text and graphic messages on the stand desktop. Module (9) is a system for accumulating statistical data in the stand event log. Module (10) is a database of template sequences of operations for successful assembly of parts. Module (11) is data export to external systems (for example, ERP, CRM, SYSLOG or others).

The example of assembly control is based on a part of the manual process of assembling a bicycle hub (hereinafter referred to as the product, assembly object), consisting of 20 different parts shown in Fig. 2. The parts are small in size, and some of the parts are visually similar to each other (Fig. 3).

To perform automated assembly control operations in real time, a structure (Fig. 5) is used, consisting of the following elements: a table, a video camera (one or more), a projector, lighting devices, a computer, a monitor, and computer program code stored in a long-term memory device of the computer. The computer and video camera, the computer and the projector are connected by switching wires.

In some embodiments, the computer and lighting fixtures are also connected by patch wires.

In some embodiments, the computer may be connected to wired and/or wireless telecommunications networks for both transmitting assembly data and receiving data (pre-trained machine learning models and/or others).

The computer program code stored in this long-term memory device ensures, when executed by one or more processors (including graphic processors): the reception of a continuous video stream, from at least one camera position above the work area (table), the entire sequence of actions described below for intelligent control of manual operations, including the determination (detection) of parts during the assembly process, detection of operations, detection of the correctness of the operator's actions at each stage, detection of the correctness of the operator's actions in assembling the entire product (assembly object).

Information about the status and correctness of the operation is displayed on the monitor screen and in the assembly work area.

In addition, data is transmitted via a software interface to external systems for accumulation and subsequent analysis.

In some cases, connected external systems can be used to voice (acoustically) inform the operator about events (errors, successful completion of a stage, lagging behind the assembly schedule, etc.).

The advantage of the method is the possibility of its use for various assembly objects, different sets of parts. Reconfiguration of the computer-implemented system implementing the proposed method is carried out by loading a new (or selecting another, already loaded) pre-trained machine vision model using machine learning methods.

The application of the proposed method can be implemented in the form of the following sequence of actions:

1. Using the interface on the monitor, the operator selects the type of product being assembled (PC) and the corresponding pre-trained machine vision model using machine learning methods.

2. The operator places all the parts on the table in the working area of the video camera capture according to the assembly instructions (Fig. 4).

3. The operator waits until the monitor displays information confirming that the set of parts on the table is correct and indicating that the assembly of the product can then begin.

4. The system, by highlighting the required part, tells the operator which part needs to be used at a given assembly step (for example, at step 1 - a bushing, at step 2 - screw No. 1, at step 3 - bearing No. 1, etc.).

5. After capturing the part, the system detects the correctness of its installation and displays a corresponding message on the monitor or in another way.

6. The system records the result of point 5 in the event log in the computer’s permanent memory.

7. The system transmits the result of point 5 to external systems via communication networks and telecommunication protocols.

8. The system evaluates the overall state of the product assembly process.

9. The system detects the use of the required tool at each stage.

10. The system detects the use of protective equipment required by safety regulations (gloves and/or helmet and/or protective clothing, etc.)

11. The system visualizes the current state of the product assembly in graphic form (Fig. 6) and records the current state in the event log. If the product assembly is not complete, the system proceeds to point 4. If all parts are exhausted, the system records information about the final result of the product assembly in the event log and proceeds to point 2.

The use of a computer-implemented method of automatic control of assembly production allows for control over the correct execution of the final set of regulated operations, identification of typical errors made during the execution of operations associated with a violation of the sequence of actions, with errors in the selection of product components (tools required for assembly and disassembly of the product), with incorrect (unintended) actions performed with the product, etc.

The claimed invention was developed within the framework of the grant (agreement No. 70-2021-00139 dated November 2, 2021), namely:

- refers to the direction of development of artificial intelligence: “Artificial intelligence for business (AI for business)”;

- refers to the event of the Research Center plan in the field of artificial intelligence: “Conducting research and development to create automatic methods for constructing data samples and accelerating modeling within the framework of tasks of intelligent control of manual operations”;

- relates to the subject of the Research Center Program in the field of artificial intelligence: "Intelligent automation of manual operations, recognition of operations in production and intelligent methods in industrial safety."

Claims (16)

1. A method for automatically monitoring assembly production, implemented using a system consisting of a module (1) for selecting an operation for assembly with an interface for the user, a module (2) for capturing images from cameras installed above the working surface of the assembly zone on the stand, a module (3) for recognizing patterns in images from cameras with the output of data on the localization and classification of objects in the images, a module (4) for filtering patterns based on a sliding window, a module (5) for detecting the assembly operation, a module (6) for a memory buffer with information on previously accumulated intermediate operations on the stand, a module (7) for assessing the state of the product assembly and the completion of intermediate operations, a module (8) for visualization and prompts for the stand operator, a module (9) for accumulating statistical data in the stand event log, a module (10) for a database of template sequences of operations for the successful assembly of parts, a module (11) for exporting data to external systems, the method includes the following steps: - the module (10) of the database of template sequences of operations for successful assembly of parts is loaded, - images are captured from video cameras using the module (2), - the operator selects the type of product to be assembled using module (2) and the corresponding pre-trained machine vision model, - the operator places the product parts on the table in the working area of the video camera capture according to the assembly instructions; - recognition of parts of the assembled product is carried out using a module (3) in images from cameras based on pre-trained machine vision models; - using module (4), filtering of recognized data is performed based on a sliding window, the size of which is selected automatically based on the speed of pattern recognition, but not less than 4 frames; - using module (5), the intermediate result of the product assembly is detected in order to determine the correctness of the operation performed; - with the help of module (6) the accumulation of data on intermediate operations occurs; - with the help of module (7), the assembly of the product and the completion of intermediate operations are assessed, while the accumulated data in module (6) and the data from module (1) about the assembled part are checked; - a signal of successful assembly of the part is generated if the sequentially accumulated data coincides with the template operations for assembling the part; - a signal about an erroneous operation is generated if the chain of operations in module (6) differs from the template. 2. The method according to item 1, additionally including visualization of the current state of the product assembly using the module (8) in graphical form. 3. The method according to item 1, additionally including recording the current state of the product assembly in the event log using module (9). 4. The method according to item 1, further comprising sending assembly data to external systems using module (11).