CN111857470B - Unattended control method and device for production equipment and controller - Google Patents
Unattended control method and device for production equipment and controller Download PDFInfo
- Publication number
- CN111857470B CN111857470B CN202010680097.0A CN202010680097A CN111857470B CN 111857470 B CN111857470 B CN 111857470B CN 202010680097 A CN202010680097 A CN 202010680097A CN 111857470 B CN111857470 B CN 111857470B
- Authority
- CN
- China
- Prior art keywords
- information
- control
- processing progress
- production equipment
- interface component
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
- G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
- G06K7/10544—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation by scanning of the records by radiation in the optical part of the electromagnetic spectrum
- G06K7/10821—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation by scanning of the records by radiation in the optical part of the electromagnetic spectrum further details of bar or optical code scanning devices
- G06K7/10861—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation by scanning of the records by radiation in the optical part of the electromagnetic spectrum further details of bar or optical code scanning devices sensing of data fields affixed to objects or articles, e.g. coded labels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Electromagnetism (AREA)
- Life Sciences & Earth Sciences (AREA)
- Toxicology (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The application relates to an unattended control method and device of production equipment and a controller. The method comprises the following steps: monitoring a control interface image of the production equipment in real time; extracting interface component information in the control interface image through an image processing algorithm; acquiring the processing progress information of the current time of the production equipment according to the interface component information; inputting a simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information; wherein the simulated operation signal corresponds to the processing progress information. The method can be used for controlling the efficiency of the production equipment.
Description
Technical Field
The application relates to the technical field of automation control, in particular to an unattended control method, an unattended control device and an unattended control controller for production equipment.
Background
The RPA (machine process automation) technology is AI, and its cognitive technology such as machine learning is very common in the automation applications of repetitive work and highly intelligent processing, such as financial processing, interface information extraction, etc. However, the application of the RPA technology in industrial manufacturing is limited, for example, the semiconductor production equipment is of various types, and the semiconductor process design and operation flow are very different, most of the existing semiconductor equipment automatic control software systems are customized for specific equipment, and the flexibility of the semiconductor equipment automatic control software system is increased by providing some custom configurations, so that the universality according to different operation flows of different equipment is difficult to achieve.
For unmanned modification of existing automation equipment, the following methods are generally adopted in the prior art:
1. after the overall control scheme of the equipment is mastered, an interface of the monitoring equipment is added;
2. the RPA professional technicians make use of the Windows series human-computer interface components and the personalized operation characteristics of the equipment to formulate a control flow.
Although the method 1 has the advantages of high degree of freedom in control and large amount of information exchange, the method has high requirement on understanding of equipment and high technical difficulty; meanwhile, customized development is required, time and labor are consumed, and the popularization is poor; and the reconstruction cost is high, the failure risk is large, and the reconstruction period is long. Although the method 2 can satisfy the advantages of most control systems of the Windows series, the method cannot satisfy OS system equipment after DOS, Linux, Unix and Windows 10; meanwhile, a professional RPA engineer is required to take charge of personalized development; and the modification period is longer. In a word, the existing unmanned modification method for the automation equipment has poor universality and high cost.
In a word, the existing unattended control method of the automation equipment needs to carry out large-scale reconstruction on the system, so that the use cost is high.
Disclosure of Invention
In view of the above, it is necessary to provide an unattended control method, apparatus and controller for a production facility, which can reduce the cost of controlling the production facility.
A method of unattended control of a production facility, the method comprising:
monitoring a control interface image of the production equipment in real time;
extracting interface component information in the control interface image through an image processing algorithm;
acquiring the processing progress information of the current time of the production equipment according to the interface component information;
inputting a simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information; wherein the simulated operation signal corresponds to the processing progress information.
In one embodiment, the interface component comprises a picture form, a feature icon or a text form; the step of obtaining the processing progress information of the current time of the production equipment according to the position of the interface component and the control character information comprises the following steps: extracting processing progress characteristic information according to the picture window, the characteristic icon or the character window; and acquiring the processing progress information of the current time of the production equipment according to the processing progress characteristic information.
In one embodiment, the interface component comprises a control button, an input box or a drop-down box; the inputting of the simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information comprises: acquiring a simulation operation signal of the production equipment at the current time according to the processing progress information; acquiring the positions of the control button, the input frame or the pull-down frame; and inputting a simulation operation signal of the production equipment at the current time according to the position of the control button, the input frame or the pull-down frame.
In one embodiment, the obtaining of the processing progress information of the current time of the production equipment according to the processing progress characteristic information includes: and comparing the processing progress characteristic information with a prestored processing schedule to acquire the processing progress information of the current time of the production equipment.
In one embodiment, the obtaining of the processing progress information of the current time of the production equipment according to the processing progress characteristic information includes: inputting the characteristic information of the processing progress into a processing progress recognition model to obtain the processing progress information of the current time of the production equipment; the processing progress recognition model is a sample set constructed according to processing progress characteristic information extracted from a picture window, a characteristic icon or a character window, is obtained through training of a preset neural network model, and is used for outputting the processing progress information of the current time of the production equipment according to the input processing progress characteristic information.
In one embodiment, the unattended control of the production facility further comprises: acquiring code scanning information of a product on the production equipment; acquiring the processing flow information of the product according to the code scanning information; and entering a control interface of the production equipment according to the processing flow information.
In one embodiment, after obtaining the processing progress information of the current time of the production equipment according to the interface component information, the method includes: and when the processing progress information corresponds to the completion of processing, controlling interface component information in the interface image according to the completion of each historical processing progress, acquiring parameters of a processing completion report, and sending the parameters to a management terminal.
In one embodiment, after extracting the interface component information in the control interface image through an image processing algorithm, the method includes: and acquiring alarm information in the interface component information, and sending the alarm information to a management terminal.
An unattended control apparatus of a production facility, the apparatus comprising:
the control interface image monitoring module is used for monitoring a control interface image of the production equipment in real time;
the image processing module is used for extracting interface component information in the control interface image through an image processing algorithm;
the processing progress determining module is used for acquiring the processing progress information of the current time of the production equipment according to the interface component information;
the input module is used for inputting simulation operation signals to the control in the interface component according to the processing progress information and the position of the control in the interface component information; wherein the simulated operation signal corresponds to the processing progress information.
An unattended controller of a production facility, the controller comprising:
the VGA module is used for collecting control interface images and providing a screen capture SDK interface;
an OCR module: the interface component information is used for identifying the interface component information in the control interface image;
a CPU module: the processing progress information of the current time of the production equipment is obtained according to the interface component information;
the device instruction simulation module: the interface component information is used for processing the control of the mouse, the keyboard and the touch screen, and inputting the processing progress information and the position of the control in the interface component information into the control in the interface component; wherein the simulated operation signal corresponds to the processing progress information;
a setting module: the device is used for setting online and offline, setting SECS communication and record, setting event parameters, setting control flow, setting a mouse, setting a keyboard and setting a touch screen;
a log module: the log is used for recording a running log and an error log;
a protocol module: for providing conditions for communicating with front and rear devices.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
monitoring a control interface image of the production equipment in real time;
extracting interface component information in the control interface image through an image processing algorithm;
acquiring the processing progress information of the current time of the production equipment according to the interface component information;
inputting a simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information; wherein the simulated operation signal corresponds to the processing progress information.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
monitoring a control interface image of the production equipment in real time;
extracting interface component information in the control interface image through an image processing algorithm;
acquiring the processing progress information of the current time of the production equipment according to the interface component information;
inputting a simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information; wherein the simulated operation signal corresponds to the processing progress information.
According to the unattended control method and device of the production equipment, the control interface image of the production equipment is monitored in real time, the control interface image of the production equipment is analyzed, the processing progress information is obtained, the simulation operation signal is input to the control in the interface component according to the processing progress information, manual control over the control interface through input equipment such as a keyboard and a mouse can be replaced, the method is strong in universality, the generation equipment does not need to be modified, and the cost is low.
Drawings
FIG. 1 is a diagram showing an example of an application environment of an unattended control method of a production apparatus;
FIG. 2 is a schematic flow chart of a method for unattended control of a production facility in one embodiment;
FIG. 3 is a schematic diagram of interface components in one embodiment;
FIG. 4 is a block diagram showing the construction of an unattended control apparatus of a production apparatus according to an embodiment;
FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The unattended control method of the production equipment can be applied to the application environment shown in fig. 1. The factory system 102 communicates with the unattended controller 104 through a network, and the unattended controller 104 communicates with the production equipment 106 through a serial port or a network port. The unattended controller 104 monitors a control interface image of the production equipment 106 in real time; extracting interface component information in the control interface image through an image processing algorithm; acquiring the processing progress information of the current time of the production equipment according to the interface component information; inputting a simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information; wherein the simulated operation signal corresponds to the processing progress information; the production equipment 106 operates according to the control information generated by the simulated operation signal. The factory System 102 includes a Manufacturing Execution System (MES) and a factory automation server, the unattended controller 104 may be, but is not limited to, various personal computers, notebook computers, etc., and the production equipment 106 may be semiconductor production equipment, semiconductor processing equipment, packaging equipment, etc. The analog operation signal is an operation signal sent by an analog keyboard, a mouse, a tablet computer and the like.
In one embodiment, as shown in fig. 2, there is provided a method of unattended control of a production facility, comprising the steps of:
and S110, monitoring a control interface image of the production equipment in real time.
In the production process of the traditional production equipment, a control system of the traditional production equipment is provided with a control interface for manually clicking and inputting parameters according to the control interface to control the operation of the equipment, for example, a parameter configuration interface appears before the operation of the control equipment, a user inputs parameters according to the parameter configuration interface, then clicks to operate, an operation result appears on the control interface in the operation process, and finally the operation is finished. Here, the unattended control apparatus monitors a control interface image of the production equipment in real time.
And S120, extracting interface component information in the control interface image through an image processing algorithm.
Wherein, the interface components include a frame form 301, a feature icon 302, a text form 303, a control button 304, an input box 305 and a drop-down box 306 (as shown in fig. 3); the frame window can be a window frame image of the control interface, such as a frame "load" including an interface title; the characteristic picture is a special icon set in the control interface, such as a warning sign "Stop"; the text window is a window containing text alternatives, such as "trends" in fig. 3; the control button is a control in the form of a button, and can be clicked, pulled and the like by simulating an operation signal, such as 'OK' in fig. 3; both the input box 305 and the drop-down box 306 may input text for selecting or setting parameters for the control device. After the input parameters are input, the input parameters are input through an analog operation signal, and the control interface image is a character window, through which whether the input parameters are correct or not can be identified, and zero error of the input parameters can be ensured. Wherein the input parameters are obtained from MES, the parameters are the current equipment processing permission and processing condition of the processed product, and the parameters comprise voltage, current, time and the like.
The image processing algorithm is used for separating the interface component from the control interface image and acquiring the position of the interface component, and comprises the identification of a picture window, a feature icon, a character window, a control button, an input box and a drop-down box, for example, the control button identification can adopt a template small image matching algorithm, each possible position in the image is traversed by the control button template small image during specific operation, whether each position is similar to the template small image or not is compared, and when the similarity is high enough, the position of the control button which is in accordance with the position is considered to be found; for the Recognition of the characters in the character window, Wang (Wang x., LIANG x., SUN l., lium.) (triangle mesh based segmentation for Chinese character understanding. in Document Analysis and Recognition (ICDAR), 201312 th International Conference (2013), IEEE, pp.1155-1159.) may be used to extract the outline of the glyph, triangulate the outline, obtain the fuzzy region and the small stroke segments according to the different features of the Triangular grid, and finally connect the small stroke segments into the complete stroke by analyzing the connectivity between the stroke segments, or extract the characters in the image by other methods.
And S130, acquiring the processing progress information of the current time of the production equipment according to the interface component information.
The processing progress of the current time can be judged by the positions, shapes and colors of a picture window, a feature icon, a character window, a control button, an input frame or a drop-down frame in the interface component information, and the processing progress information of the current time of the production equipment can be obtained. For example, if the frame window is in the start interface and the text window includes the text "start", the processing progress is to start processing; when a starting control button appears, judging that the processing progress is that an operation signal needs to be input to control the production equipment to start processing; of course, the processing progress may also be determined by the feature icon and the text, for example, if the control text information is "step 1", the processing progress is 1, and if the control text information is "step 2", the processing progress is 2, and the processing progress may be obtained by analyzing the control text information at some special positions (e.g., title positions).
S140, inputting a simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information; wherein the simulated operation signal corresponds to the processing progress information.
The analog operation signal is an operation signal sent by an analog keyboard, a mouse, a tablet computer and the like. Each processing progress corresponds to different control modes, for example, when the processing progress is 1, configuration parameters need to be input into a character window, preset configuration parameters are obtained through a script file according to the position of the identified character window, the character window is input, when the processing progress is 2, control equipment starts to operate, and according to the position of a start control button of a control interface, a mouse is simulated to send a click control instruction so as to click the start control button. The processing progress information comprises processing time of the production equipment, different processing time may need different control operation, and an analog operation signal is input according to the processing time.
According to the unattended control method of the production equipment, the control interface image of the production equipment is monitored in real time, the control interface image of the production equipment is analyzed, the processing progress information is obtained, the simulation operation signal is input to the control in the interface assembly according to the processing progress information, manual operation can be replaced, and the control interface can be controlled through input equipment such as a keyboard and a mouse.
In one embodiment, the interface component comprises a picture form, a feature icon or a text form; the step of obtaining the processing progress information of the current time of the production equipment according to the position of the interface component and the control character information comprises the following steps: extracting processing progress characteristic information according to the picture window, the characteristic icon or the character window; and acquiring the processing progress information of the current time of the production equipment according to the processing progress characteristic information.
The processing progress characteristic information is image characteristic information about a picture window, a characteristic icon or a character window, namely, images of the picture window, the characteristic icon or the character window are processed into image characteristic data, and information and meaning in the images can be determined according to different image characteristic data. For example, when the text window includes "List View", it is determined that the product model needs to be selected, and an analog operation signal for selecting the product model from the List needs to be transmitted.
In one embodiment, the interface component comprises a control button, an input box or a drop-down box; the inputting of the simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information comprises: acquiring a simulation operation signal of the production equipment at the current time according to the processing progress information; acquiring the positions of the control button, the input frame or the pull-down frame; and inputting a simulation operation signal of the production equipment at the current time according to the position of the control button, the input frame or the pull-down frame.
In one embodiment, the obtaining of the processing progress information of the current time of the production equipment according to the processing progress characteristic information includes: comparing the processing progress characteristic information with a prestored processing schedule according to the processing progress characteristic information to obtain the processing progress information of the current time of the production equipment; and each piece of processing progress information in the processing progress chart is associated and matched with the processing progress characteristic information one by one.
In one embodiment, the obtaining of the processing progress information of the current time of the production equipment according to the processing progress characteristic information includes: inputting the characteristic information of the processing progress into a processing progress recognition model to obtain the processing progress information of the current time of the production equipment; the processing progress recognition model is a sample set constructed according to processing progress characteristic information extracted from a picture window, a characteristic icon or a character window, is obtained through training of a preset neural network model, and is used for outputting the processing progress information of the current time of the production equipment according to the input processing progress characteristic information.
The preset neural network model comprises neural networks such as LeNet-5, AlexNet, ZFNET, VGG-16, GoogLeNet, ResNet and the like, and of course, the preset neural network model also comprises a logistic regression function. The images of interface components such as a picture window, a feature icon, a character window, a control button, an input frame, a drop-down frame and the like of the production equipment can be collected during manual operation, and a preset neural network model is trained.
In one embodiment, the unattended operation control method of the production equipment further comprises: acquiring code scanning information of a product on the production equipment; acquiring the processing flow information of the product according to the code scanning information; and entering a control interface of the production equipment according to the processing flow information. The scanning information includes two-dimensional code information or bar code information, and of course, the scanning information may also be information of codes in other forms. After all the progresses of a certain processing flow of a product on the production equipment are completed, a code pattern can be formed, the product or the product package is pasted on the surface of the product or the product package after being printed, and code scanning information is obtained through scanning of the code scanning equipment. For example, after the product M completes the processing flow a, the processing flow a is printed in a two-dimensional code pattern and is pasted on the product M or the package of the product M, information that the product M has completed the processing flow a is obtained through the package two-dimensional code image of the product M or the product M, then the system enters the processing flow B according to the completion condition of the processing flow a, and the production equipment is controlled to enter a control interface of the processing flow B. In this implementation, all form to every flow of working and sweep a yard information and supply to sweep a yard equipment scanning, can judge automatically which processing stage the product has got into, guarantee the automation of product processing and go on, especially to the product that artifical naked eye can't judge processing stage, can guarantee through this method that each processing stage of product goes on in proper order.
In one embodiment, after obtaining the processing progress information of the current time of the production equipment according to the interface component information, the method includes: and when the processing progress information corresponds to the completion of processing, controlling interface component information in the interface image according to the completion of each historical processing progress, acquiring parameters of a processing completion report, and sending the parameters to a management terminal. The control interface image of the real-time monitoring production equipment can extract interface component information in the control interface image, store parameters in the processing progress at any time, input report parameters into a parameter input box of a processing completion report according to the stored parameters, and then send the report parameters to the management terminal. In the embodiment, various parameter filling in the processing completion report can be automatically completed, so that manual intervention is avoided, and the accuracy of filling in the report is ensured.
In one embodiment, after extracting the interface component information in the control interface image through an image processing algorithm, the method includes: and acquiring alarm information in the control interface image and sending the alarm information to a management terminal. The management terminal is a mobile phone, a personal computer or a tablet computer used by a manager. In this embodiment, in the unattended environment, through sending alarm information to management terminal, guarantee to inform artifical the processing under the unusual circumstances appearing.
In one embodiment, after obtaining the processing progress information of the current time of the production equipment according to the interface component information, the method includes: and when the processing progress information of the current time of the production equipment is the starting information, sending the processing starting information to the MES. In this embodiment, the MES processing start time can be notified by sending the processing start information to the MES.
In one embodiment, after obtaining the processing progress information of the current time of the production equipment according to the interface component information, the method includes: and when the processing progress information of the current time of the production equipment is end information, sending processing end information to the MES system. In this embodiment, the MES processing end time can be notified by sending the processing end information to the MES.
It should be understood that, although the steps in the flowchart of fig. 2 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 2 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
In one embodiment, as shown in fig. 4, there is provided an unattended control apparatus of a production apparatus, including: a control interface image monitoring module 210, an image processing module 220, a processing progress determining module 230 and an input module 240, wherein:
and the control interface image monitoring module 210 is used for monitoring the control interface image of the production equipment in real time.
And the image processing module 220 is configured to extract interface component information in the control interface image through an image processing algorithm.
And a processing progress determining module 230, configured to obtain, according to the interface component information, processing progress information of the current time of the production equipment.
The input module 240 is configured to input a simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information; wherein the simulated operation signal corresponds to the processing progress information.
In one embodiment, the interface component comprises a frame, a feature icon or a text frame. The image processing module 220 includes: the characteristic extraction unit is used for extracting processing progress characteristic information according to the picture window, the characteristic icon or the character window; and the processing progress information acquisition unit is used for acquiring the processing progress information of the current time of the production equipment according to the processing progress characteristic information.
In one embodiment, the interface component includes a control button, an input box, or a drop-down box. The input module 240 includes: the simulation operation signal acquisition unit is used for acquiring a simulation operation signal of the production equipment at the current time according to the processing progress information; the position acquisition unit is used for acquiring the positions of the control buttons, the input frame or the pull-down frame; and the input unit is used for inputting the simulation operation signal of the production equipment at the current time according to the position of the control button, the input frame or the pull-down frame.
In one embodiment, the processing progress determining module 230 is further configured to compare the processing progress characteristic information with a pre-stored processing schedule to obtain the processing progress information of the current time of the production equipment.
In one embodiment, the processing progress determining module 230 is further configured to input the processing progress characteristic information into a processing progress identification model, so as to obtain the processing progress information of the current time of the production equipment; the processing progress recognition model is a sample set constructed according to processing progress characteristic information extracted from a picture window, a characteristic icon or a character window, is obtained through training of a preset neural network model, and is used for outputting the processing progress information of the current time of the production equipment according to the input processing progress characteristic information.
In one embodiment, the unattended operation control device of the production equipment further comprises: the code scanning information acquisition module is used for acquiring code scanning information of the product on the production equipment; the processing flow information acquisition module is used for acquiring the processing flow information of the product according to the code scanning information; and the control interface entering module is used for entering a control interface of the production equipment according to the processing flow information.
In one embodiment, the unattended operation control device of the production equipment further comprises: and the report parameter input module is used for controlling the interface component information in the interface image according to the completion of each historical processing progress when the processing progress information corresponds to the completion of processing, acquiring the parameters of the processing completion report and sending the parameters to the management terminal.
In one embodiment, the unattended operation control device of the production equipment further comprises: and the alarm information sending module is used for acquiring the alarm information in the control interface image and sending the alarm information to the management terminal.
For specific limitations of the unattended control device of the production equipment, reference may be made to the above limitations of the unattended control method of the production equipment, and details thereof are not repeated here. The modules in the unattended control device of the production equipment can be completely or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, there is provided an unattended controller of a production apparatus, comprising: the VGA module is used for collecting control interface images and providing a screen capture SDK interface; an OCR module: the interface component information is used for identifying the interface component information in the control interface image; a CPU module: the processing progress information of the current time of the production equipment is obtained according to the interface component information; the device instruction simulation module: the interface component information is used for processing the control of the mouse, the keyboard and the touch screen, and inputting the processing progress information and the position of the control in the interface component information into the control in the interface component; wherein the simulated operation signal corresponds to the processing progress information; a setting module: the device is used for setting online and offline, setting SECS communication and record, setting event parameters, setting control flow, setting a mouse, setting a keyboard and setting a touch screen; a log module: the log is used for recording a running log and an error log; a protocol module: for providing conditions for communicating with front and rear devices.
The unattended controller is a standardized programmable automation control component series based on technologies such as RPA (robust Process Automation), image processing, keyboard/mouse simulation and the like. The RPA technology is an automatic application of cognitive technologies such as AI and machine learning in repetitive work and highly intelligent processing, is widely applied to aspects such as financial processing and interface information extraction, and has certain limitation in the application of industrial manufacturing. The RPA technology in the industrial manufacturing field is most part cognitive and operational for Winform window controls, but early applications of DOS and other non-Microsoft series environments are yet to be developed.
The setting module comprises a dictionary setting tool, an OCR recognition setting tool, a mouse setting tool, a touch screen setting tool and a control flow setting tool.
The dictionary setting tool is used for setting mouse, keyboard, image recognition configuration program code and name. The mouse setting tool is used for setting the movement and click instructions of a mouse in the equipment instruction simulation module, and the interface type of the mouse in the equipment instruction simulation module is a PS2 interface or a USB interface; the setting of the PS2 interface comprises setting of single relative movement amount and setting of movement times; setting single relative movement amount including setting X coordinate and setting Y coordinate; setting the X coordinate to move leftwards to be a negative number and rightwards to be a positive number; and setting the Y coordinate to move leftwards to be a negative number and rightwards to be a positive number. Further comprising the following steps of setting the resolution of the USB screen: inputting the actual resolution on the X axis and the Y axis, and after clicking the setting, re-plugging the USB interfaces at the lower two ends to take effect.
The keyboard setting tool is used for setting the instruction of the keyboard in the equipment instruction simulation module, and the input instruction types of the keyboard in the equipment instruction simulation module comprise character types, command types and combination key types; the character type is used for simulating a keyboard to input character keys; the command type is used for simulating keyboard input function keys and control keys; the combination key type is used for simulating a keyboard to input combination keys.
The touch screen setting tool is used for setting clicking and moving instructions of the touch screen in the equipment instruction simulation module.
The OCR recognition setting tool is used for setting an image recognition interface, and the type of the image recognition interface is a template matching interface, a character recognition interface or a template matching positioning interface; the template matching interface is used for inputting XY coordinates and width and height of the area where the template is located, wherein the range of the XY coordinate area is larger than that of the template; the template matching and positioning interface is used for matching according to the template and returning the matched coordinates; the character recognition interface is used for inputting XY coordinates and width and height of an area where characters to be recognized are located.
The control flow setting tool is used for forming a set of flows of operation, monitoring and reporting according to the characteristics of different application devices.
By utilizing the unattended controller, the invention also provides an unattended control system for factory production management, and the following three system architectures can be used for meeting the industrial manufacturing field according to the practical application environment.
Example 1: and (4) an interface component control method.
The interface component control method is a remote control method for cognition and operation based on a Winform window, an unattended intelligent controller is additionally arranged outside, meanwhile, interface component control software is required to be installed on an equipment control computer, the architecture is shown in figure 2, the remote control method comprises the unattended controller, an instruction system and controlled equipment, the instruction system comprises an MES system and an EAP automatic server, the controlled equipment is an industrial personal computer, the EAP automatic server is in data communication with the unattended controller through an HSMS protocol and an SECS-I protocol, the industrial personal computer is required to be pre-installed with the interface component control software, and the interface component control software is in serial port or internet access communication with the unattended controller.
Example 2: and (4) an image cognition control method.
An image cognition control method is an external control method based on cognition technique, image display technique, computer accessory control technique and deep learning theory, except that part of computer standard peripheral accessories (display, keyboard and mouse) are connected, it does not affect the operation of original control box system, and can be extensively used in various control equipments of DOS series, Windows series and other non-Microsoft series, its structure is shown in figure 3, it includes unattended controller, instruction system and controlled equipment, the instruction system includes MES system and EAP automation server, the controlled equipment is industrial computer, EAP automation server makes data communication with unattended controller by HSMS protocol and SECS-I protocol, unattended controller makes data communication with industrial computer by SECS standard, concretely, host computer of unattended controller and industrial computer, VGA signals and keyboard signals are communicated with each other between the screen and the keyboard.
Example 3: hybrid control method
The hybrid control method is a combination of an interface component control method and an image cognition control method, and is based on the interface component control method, a small part of nonstandard components are supplemented and monitored by image cognition, specifically, as shown in fig. 4, the hybrid control method comprises an unattended controller, an instruction system and controlled equipment, wherein the instruction system comprises an MES system and an EAP automation server, the controlled equipment is an industrial personal computer, the EAP automation server performs data communication with the unattended controller through an HSMS protocol and an SECS-I protocol, interface component control software needs to be installed on the industrial personal computer in advance, serial ports or internet interfaces are adopted for communication between the interface component control software and the unattended controller, meanwhile, the unattended controller performs data communication with the unattended controller through an SECS standard, and specifically, image display signals are transmitted between a screen of the industrial personal computer and a host.
The image cognitive control method in embodiment 2 is most versatile, and can be widely applied to various control devices of various DOS series, Windows series, and other non-Microsoft series, and is an optimal scheme.
Taking embodiment 2 as an example, the operation example of the unattended operation control method of the production equipment provided by the invention is as follows: the MES system sends an instruction to a plant equipment group, for example, sends a control instruction of a process flow 1 of a product A, the instruction is transmitted to the unattended controller after passing through the EAP automation server, and the unattended controller starts to work after receiving the instruction, specifically: starting a controller of the generation equipment according to the processing flow 1, entering a control interface of the production equipment, and monitoring a control interface image of the production equipment in real time; extracting interface component information in the control interface image through an image processing algorithm; acquiring the processing progress information of the current time of the production equipment according to the interface component information; inputting a simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information; wherein the simulated operation signal corresponds to the processing progress information.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing control interface image data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of unattended control of a production apparatus.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for unattended control of a production facility, the method comprising:
monitoring a control interface image of the production equipment in real time;
extracting interface component information in the control interface image through an image processing algorithm;
acquiring the processing progress information of the current time of the production equipment according to the interface component information;
inputting a simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information; the simulation operation signal corresponds to the processing progress information, the simulation operation signal is an operation signal sent by a simulation keyboard, a mouse and a touch screen, and the simulation operation signal comprises the steps of inputting configuration parameters into a character window, clicking a control button and making option selection according to a list.
2. The method of claim 1, wherein the interface components include screen forms, feature icons;
the obtaining of the processing progress information of the production equipment at the current time according to the interface component information includes:
extracting processing progress characteristic information according to the picture window and the characteristic icons;
and acquiring the processing progress information of the current time of the production equipment according to the processing progress characteristic information.
3. The method of claim 1, wherein the interface component comprises a control button, an input box, or a drop-down box;
the inputting of the simulation operation signal to the control in the interface component according to the processing progress information and the position of the control in the interface component information comprises:
acquiring a simulation operation signal of the production equipment at the current time according to the processing progress information;
acquiring the positions of the control button, the input frame or the pull-down frame;
and inputting a simulation operation signal of the production equipment at the current time according to the position of the control button, the input frame or the pull-down frame.
4. The method according to claim 2, wherein the obtaining of the processing progress information of the current time of the production equipment according to the processing progress characteristic information comprises:
and comparing the processing progress characteristic information with a prestored processing schedule to acquire the processing progress information of the current time of the production equipment.
5. The method according to claim 2, wherein the obtaining of the processing progress information of the current time of the production equipment according to the processing progress characteristic information comprises:
inputting the characteristic information of the processing progress into a processing progress recognition model to obtain the processing progress information of the current time of the production equipment; the processing progress recognition model is a sample set constructed according to processing progress characteristic information extracted from a picture window and a characteristic icon, is obtained through training of a preset neural network model, and is used for outputting the processing progress information of the current time of the production equipment according to the input processing progress characteristic information.
6. The method of claim 1, further comprising:
acquiring code scanning information of a product on the production equipment;
acquiring the processing flow information of the product according to the code scanning information;
and entering a control interface of the production equipment according to the processing flow information.
7. The method of claim 1, after acquiring the processing progress information of the current time of the production equipment according to the interface component information, comprising:
and when the processing progress information corresponds to the completion of processing, controlling interface component information in the interface image according to the completion of each historical processing progress, acquiring parameters of a processing completion report, and sending the parameters to a management terminal.
8. The method of claim 1, after extracting interface component information in the control interface image through an image processing algorithm, comprising:
and acquiring alarm information in the interface component information, and sending the alarm information to a management terminal.
9. An unattended control apparatus of a production facility, the apparatus comprising:
the control interface image monitoring module is used for monitoring a control interface image of the production equipment in real time;
the image processing module is used for extracting interface component information in the control interface image through an image processing algorithm;
the processing progress determining module is used for acquiring the processing progress information of the current time of the production equipment according to the interface component information;
the input module is used for inputting simulation operation signals to the control in the interface component according to the processing progress information and the position of the control in the interface component information; the simulation operation signal corresponds to the processing progress information, the simulation operation signal is an operation signal sent by a simulation keyboard, a mouse and a touch screen, and the simulation operation signal comprises the steps of inputting configuration parameters into a character window, clicking a control button and making option selection according to a list.
10. An unattended controller of a production facility, the controller comprising:
a VGA module: the system is used for acquiring control interface images and providing a screen capture SDK interface;
an OCR module: the interface component information is used for identifying the interface component information in the control interface image;
a CPU module: the processing progress information of the current time of the production equipment is obtained according to the interface component information;
the device instruction simulation module: the interface component information processing device is used for simulating and sending operation signals sent by a mouse, a keyboard and a touch screen, and inputting simulated operation signals to the control in the interface component according to the processing progress information and the position of the control in the interface component information; the simulation operation signal corresponds to the processing progress information, and comprises the steps of inputting configuration parameters into a character window, clicking a control button and making option selection according to a list;
a setting module: the device is used for setting online and offline, setting SECS communication and record, setting event parameters, setting control flow, setting a mouse, setting a keyboard and setting a touch screen;
a log module: the log is used for recording a running log and an error log;
a protocol module: for providing conditions for communicating with front and rear devices.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010680097.0A CN111857470B (en) | 2020-07-15 | 2020-07-15 | Unattended control method and device for production equipment and controller |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010680097.0A CN111857470B (en) | 2020-07-15 | 2020-07-15 | Unattended control method and device for production equipment and controller |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111857470A CN111857470A (en) | 2020-10-30 |
| CN111857470B true CN111857470B (en) | 2022-02-22 |
Family
ID=72984478
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010680097.0A Active CN111857470B (en) | 2020-07-15 | 2020-07-15 | Unattended control method and device for production equipment and controller |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111857470B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112712001A (en) * | 2020-12-25 | 2021-04-27 | 中核核电运行管理有限公司 | Intelligent response system and method for nuclear power plant control room alarm |
| CN113380000B (en) * | 2021-04-30 | 2022-12-16 | 海纳云物联科技有限公司 | Video-based fire alarm processing operation monitoring method for fire alarm controller |
| CN113190157B (en) * | 2021-05-24 | 2024-06-14 | 浙江热刺激光技术有限公司 | Control method, device and equipment of processing equipment and storage medium |
| CN113902902A (en) * | 2021-09-30 | 2022-01-07 | 东莞华贝电子科技有限公司 | Test method, device, system and storage medium |
| CN115718670B (en) * | 2023-01-06 | 2023-04-28 | 无锡芯享信息科技有限公司 | Automatic substitution operating system and method for semiconductor dust-free room |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107370823A (en) * | 2017-08-10 | 2017-11-21 | 南京泰治自动化技术有限公司 | Data acquisition and long-range control method, device and computer-readable recording medium |
| CN108509109A (en) * | 2018-03-30 | 2018-09-07 | 平安科技（深圳）有限公司 | Billing information input method, device, computer equipment and storage medium |
| KR20190106369A (en) * | 2018-03-09 | 2019-09-18 | 타이아(주) | Programmable Logic Controller pattern analysis system for improving lifetime and productivity of factory automation facilities, and method thereof |
| CN111144401A (en) * | 2019-11-06 | 2020-05-12 | 华能国际电力股份有限公司海门电厂 | Touch screen control operation method for deep learning and visual servo of power plant centralized control room |
| CN111290955A (en) * | 2020-02-18 | 2020-06-16 | 卡斯柯信号有限公司 | Non-invasive automatic test system for subway signal system test software |
-
2020
- 2020-07-15 CN CN202010680097.0A patent/CN111857470B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107370823A (en) * | 2017-08-10 | 2017-11-21 | 南京泰治自动化技术有限公司 | Data acquisition and long-range control method, device and computer-readable recording medium |
| KR20190106369A (en) * | 2018-03-09 | 2019-09-18 | 타이아(주) | Programmable Logic Controller pattern analysis system for improving lifetime and productivity of factory automation facilities, and method thereof |
| CN108509109A (en) * | 2018-03-30 | 2018-09-07 | 平安科技（深圳）有限公司 | Billing information input method, device, computer equipment and storage medium |
| CN111144401A (en) * | 2019-11-06 | 2020-05-12 | 华能国际电力股份有限公司海门电厂 | Touch screen control operation method for deep learning and visual servo of power plant centralized control room |
| CN111290955A (en) * | 2020-02-18 | 2020-06-16 | 卡斯柯信号有限公司 | Non-invasive automatic test system for subway signal system test software |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111857470A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111857470B (en) | Unattended control method and device for production equipment and controller | |
| JP6551565B2 (en) | Process analysis apparatus, process analysis method, and process analysis program | |
| CN109407544B (en) | System module of simulation machine operation picture of non-invasive data extraction system | |
| KR102543064B1 (en) | System for providing manufacturing environment monitoring service based on robotic process automation | |
| CN113379399B (en) | RPA component recommendation method based on state transition probability model | |
| US20210182101A1 (en) | Program generating device, program generating method, and information storage medium | |
| CN109815434A (en) | Page editing method, apparatus, equipment and storage medium | |
| WO2023044631A1 (en) | A device, system, method and storage medium for ai application deployment | |
| JP2019169044A (en) | Software robot definition information generation system, software robot definition information generation method, and program | |
| JPH0887378A (en) | Command execution system by mouse operation recognition | |
| KR100307616B1 (en) | Industrial process monitoring and controlling system | |
| Park et al. | A reverse engineering approach to generate a virtual plant model for PLC simulation | |
| CN115648202A (en) | Robot visual perception and reaction planning combined method and system based on sequential logic | |
| US20200066016A1 (en) | System For Editing Monitor Script In Screen Image And Method Thereof | |
| CN109508953B (en) | Form template generation method | |
| CN117573006B (en) | Method and system for batch pick-up of RPA screen interface elements | |
| CN112908328A (en) | Equipment control method, system, computer equipment and storage medium | |
| CN109213101B (en) | Method and system for preprocessing under robot system | |
| CN114895884B (en) | Method and system for writing and executing RPA script of terminal equipment | |
| WO2021087818A1 (en) | Method, apparatus and system for capturing knowledge in software | |
| WO2022234678A1 (en) | Machine learning device, classification device, and control device | |
| CN114281627A (en) | Automatic operation method and device for recovery detection | |
| Pan et al. | Research on functional test of mobile app based on robot | |
| CN114453852A (en) | Method and system for controlling mechanical arm to assemble blade based on voice recognition | |
| CN115495057B (en) | Method and system for realizing windows and HDFS communication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |