JP5084798B2 - Application state recognition method, apparatus and program - Google Patents

Description

  The present invention relates to a technique for supporting a user who performs an operation smoothly through an application (AP) running on a computer so that the user can smoothly perform the operation.

  To support user operations, it is necessary to know the current user and application status. This is because if the current state is not known, the general support can be applied regardless of the state, and specific and individual support suitable for each situation cannot be performed.

  As a conventional method of knowing the state of a user or an application, there is a method of using information provided by an application program or an OS. For example, in Windows (registered trademark) which is an OS made by Microsoft (registered trademark), tools such as “AutoIt” and “Spy ++” are known. With these tools, it is possible to acquire basic information (for example, title, class, size, etc.) of a window on the OS, and to acquire a message exchanged between the window and process and the OS. Therefore, by combining these pieces of information in a composite manner, it is possible to acquire the application state at a certain level.

"AutoIt Script Home Page", [online], [Search August 13, 2009], URL <https://www.autoitscript.com/> "Home Page: Spy ++", [online], [Search August 13, 2009], URL <https://msdn.microsoft.com/en-us/library/aa264396 (VS.60) .aspx>

  However, the above method has a problem that information necessary for user support cannot be acquired sufficiently depending on the environment operated by the user. For example, in a thin client environment that is often introduced for the purpose of enhancing security, in many cases, only very limited information can be acquired. In a method of transferring relatively popular screen information as a thin client mounting method, it is not possible to acquire sufficient information of an application using a conventional method.

  The present invention has been made in view of the above circumstances, and the object of the present invention is to operate the application even in an environment where conventional methods cannot acquire sufficient information from the application or the OS. The object is to provide a system capable of recognizing a state.

  In order to achieve the above object, the present invention provides an application state recognition device for recognizing an operation state of an application running on a computer, and obtaining image data that obtains information output from the application to a display device as image data. Means, preprocessing means for performing preprocessing such as noise removal and normalization on the acquired image data, feature quantity extraction means for extracting feature quantities of the preprocessed image data, and feature quantities of the image data A learning unit for previously storing learning data for identifying the operation state of the application as an identification dictionary; and an identification unit for identifying the operation state of the application from the identification dictionary based on the extracted feature amount. To do.

  According to the present invention, since the operation state of the application is identified by performing image recognition on the image data displayed on the display device by the operation of the application, a special device such as a camera is used as in the conventional method. There is no need to prepare separately or perform complicated initial settings. Therefore, the operation state of the application can be acquired under various environments as a general technique.

  Further, as a preferred aspect of the present invention, the pre-processing unit may replace the variable area of the acquired image data with a predetermined fixed value or fixed pattern for the acquired image data. . That is, if image recognition is simply performed on image data, a difference due to a small operation state is detected too much and misrecognition is likely to occur. Specifically, the input image and the image in the identification dictionary are variable depending on the operation state. The image portion (variable region) is recognized after being filled with a uniform pattern. This makes it possible to more reliably recognize the operation state of the application ignoring the difference on the image caused by the data input state.

  As described above, according to the present invention, the current operating state of an application can be grasped by image recognition for image data on a screen without requiring any special equipment or setting. In addition, in the conventional technology, the difference in the operating state of the application appears in the difference on the image, and it is more reliable by removing the difference in the image state due to the state that can be ignored in the preprocessing stage. Recognized. As a result, various applications such as a proposal of the next operation by the user, recovery from an error state, and autonomous operation are possible, and the application operation by the user can be supported more strongly.

Configuration diagram of application state recognition system Flow chart explaining operation of application state recognition system Flow chart explaining operation of user support application Diagram explaining variable area replacement processing Diagram explaining variable area replacement processing

  An embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a configuration diagram of an application state recognition system of the present invention. In the application state recognition system according to the present embodiment, a service for recognizing and providing the operation state of another application is provided to the user support application. That is, as shown in FIG. 1, a plurality of existing applications AP1, AP2,. Each existing application AP1, AP2 uses an input device 15 such as a keyboard and a mouse and a display 16 as a display device as a user interface for the user 1.

  The user support application 20 is an application that supports the operation of the user 1 who operates the existing applications AP1, AP2,. Various forms of the user support application 20 are conceivable. For example, the user support application 20 proposes the next operation based on the state of the existing applications AP1, AP2,... On the current screen, or the user 1 is determined to be in an error state. Such as adding a function to perform autonomous operation while judging the situation in an automatic operation system. The application state recognition system 100 according to the present invention recognizes the operation state of the existing applications AP1, AP2,... With respect to the user support application 20, and provides the recognition result.

  The application state recognition system 100 will be described in detail below. The application state recognition system 100 according to the present invention has the same configuration as that of a general image recognition system. As shown in FIG. 1, the image acquisition unit 110, the preprocessing unit 120, the feature extraction unit 130, the identification Unit 140 and identification dictionary 150. The application state recognition system 100 according to the present invention can also be realized by a computer and a program, and can be recorded on a recording medium or provided through a network. Furthermore, the operating environment and mounting form of the application state recognition system 100 and the user support application 20 according to the present invention are not questioned. That is, it may operate on the same computer as the computer on which the existing applications AP1, AP2,... Operate, or may operate on another computer such as a thin client environment. Furthermore, each part of the application state recognition system 100 may be distributed and implemented in a plurality of computers instead of a single computer.

  The image acquisition unit 110 acquires, as image data, a window, an entire screen, or the like displayed on the display 16 using functions provided by the OS 11 or a system library associated therewith.

  The preprocessing unit 120 performs processing for facilitating recognition processing on image data. Specifically, processing such as binarization performed to remove extraneous noise as much as possible at the time of recognition or normalization that aligns the image size to a constant value is performed. Further, the pre-processing unit 120 performs variable region replacement processing specified from the image data as necessary. The variable area and replacement process will be described later.

  In order to recognize the image data, the feature extraction unit 130 extracts a feature amount representing the image data with a smaller number of information as a feature vector. Regarding the method for calculating the feature amount, since there are abundant existing methods that have been studied in various forms, they are used. Specifically, various feature amounts such as SIFT feature amount, HoG feature amount, and Haar-like feature amount can be used. Further, when the template matching method is used in the identification unit 140 described later, if appropriate preprocessing such as normalization of the image size is performed in the preprocessing unit 120, the image data is used as a feature amount as it is.

  The identification unit 140 refers to information (learning data) in the identification dictionary 150 created by storing information learned in advance, and identifies and classifies the input image based on the feature vector of the input image. Identification / classification is processing for determining which of the classes indicating the state to be recognized the input image belongs to. Prepare as many classes as you want to distinguish and recognize. Normally, all data in the identification dictionary 150 is divided into several finite classes based on the feature vectors. Then, the class to which the input image belongs is identified by comparing the feature vector representing the class (calculated separately) or the feature vector of all data in the dictionary with the feature vector of the input image. When comparing, there are various existing methods, such as selecting a vector that minimizes the Euclidean distance, using a neural network, using a support vector machine, and using a template matching method. To do.

  Next, the operation of the application state recognition system according to the present invention will be described with reference to FIG.

  As shown in FIG. 2, first, the image acquisition unit 110 acquires image data displayed on the display 16 in response to a request from a service providing destination application (the user support application 20 in the present embodiment) or the like. (Step S1). Note that the request may specify the existing applications AP1, AP2,. Next, the preprocessing unit 120 performs image processing as preprocessing so as to facilitate image recognition in the subsequent identification processing on the image data (step S2). Next, the feature extraction unit 130 extracts a feature vector as a feature amount from the preprocessed image data (step S3). Then, the identification unit 140 compares the extracted feature quantity with the data in the identification dictionary 150 to determine the class to which the image data belongs, and returns the class to the service providing destination application as the application operating state ( Step S4).

  Next, the operation of the user support application using the application state recognition system according to the present invention will be described with reference to FIG. Here, the user support application 20 will be described by taking as an example an application that assists the user 1 who operates the existing applications AP1, AP2,.

  As shown in FIG. 3, the user support application 20 designates image areas (windows, etc.) of existing applications AP1, AP2,... Recognition is requested (step S11). The application state recognition system 100 identifies and responds to the operation states of the existing applications AP1, AP2,... According to the operation described above with reference to FIG. The user support application 20 performs user support processing such as support for the next operation based on the acquired operation state (step S13). Since the user support process is the same as that conventionally known, detailed description thereof is omitted.

  By the way, in the present invention, as a means for grasping the operation state of the existing applications AP1, AP2,..., A method of analyzing the image data on the display screen by an image recognition technique is employed. However, simply performing image recognition on a screen image causes excessive detection of a difference due to an input situation in a text area, and there is a problem that the possibility of erroneous recognition increases.

  Therefore, in the present invention, the preprocessing unit 120 performs a process of replacing the variable area with a predetermined fixed value or fixed pattern as necessary. Here, the “variable area” means an area that has the same class to which the application operating state belongs, but whose contents may change depending on the operating state. For example, it corresponds to a text area for address input on the address input screen.

  Specifically, if the pre-processing unit 120 does not want to distinguish between different input character strings or different selection statuses such as radio buttons, the pre-processing unit 120 defines a variable region that does not want to distinguish between minor state differences. Fill with color or pattern. After that, by calculating the feature amount, it becomes possible to recognize with high accuracy ignoring the state difference such as the character string difference.

  Furthermore, if you want to distinguish between multiple states, such as text input and text not input, if some text is input, you can distinguish it by replacing it with a fixed character string indicating that text is being input. It is. Even when an image is drawn in the variable area, a technique such as replacement with a specific image can be applied. Further, for example, there is a case where it is necessary to fill a variable area in an uninput image with a fixed pattern indicating non-input. By applying this method, it is possible to distinguish and recognize any plurality of states.

  FIG. 4 shows an example of replacement processing in the preprocessing unit. FIG. 4 shows a window having a text area, a radio button, and a variable graphic area as a specific window of an application to identify a class to which the window belongs (input window such as name and address) regardless of the input status. FIG. 6 is a diagram showing an outline of a pre-processing example applied to an image, and shows before and after processing. The processing content is a process of filling a variable area with a fixed color such as a background color or a fixed pattern.

  In order to perform such processing, it is necessary to designate a variable region for which recognition accuracy is to be controlled. One method is a method of recognizing a necessary variable area by performing image recognition similar to the state recognition flow shown in FIG. For detection of a rectangular region of a character string, an existing method such as character recognition using a Fourier descriptor can be applied. In addition, even a general image can be recognized by using an existing SIFT feature value or the like and detecting an image having the same feature value as an image showing a designated variable region.

  However, since the above method cannot predict what kind of image will appear, it is not preferable for general-purpose application. Therefore, one of general-purpose methods will be described with reference to FIG. FIG. 5 illustrates an example of variable area detection.

  In the detection method shown in FIG. 5, when creating learning data of the identification dictionary 150 in order to detect general variable areas such as character areas and image areas, the variable areas in the learning target image are manually designated. The variable area is filled and converted into a fixed area, the feature amount is calculated, and both the converted and unconverted areas are stored. At that time, the upper left corner of the window or screen of each converted fixed area (assuming that it is rectangular) is the origin, and the coordinates of the upper left corner and the lower right corner are recorded as the coordinates indicating the variable area together with the image feature amount. Keep it. The preprocessing unit 120 replaces the area represented by the coordinates of the variable area of each data recorded in the identification dictionary 150 with the fixed area for the input image. The feature extraction unit 130 calculates a feature amount for each piece of image data after replacement. Then, the identification unit 140 compares each feature amount with each data in the identification dictionary 150, and uses the operation state indicated by the most similar feature amount as the identification result. However, as a condition for applying this method, there is a condition that the size of the image is the same during learning and during recognition.

  In addition, as a method that can be applied more generally, not only the image for learning but also the input image, the variable region is manually specified, the variable region is filled and converted into the fixed region, and then the feature amount is set. There is a method of calculating and recognizing.

  As described above, the application state recognition system according to the present invention recognizes the operation state of the application based on the image data of the display screen displayed by the existing applications AP1, AP2,. This makes it possible to perform recognition processing under various operating environments. In addition, in the conventional technology, the difference in the operating state of the application appears in the difference on the image, and it is more reliable by removing the difference in the image state due to the state that can be ignored in the preprocessing stage. Recognized. As a result, various applications such as a proposal of the next operation by the user, recovery from an error state, and autonomous operation are possible, and the application operation by the user can be supported more strongly.

  DESCRIPTION OF SYMBOLS 1 ... User, 10 ... Computer, 11 ... OS, 15 ... Input device, 16 ... Display, 20 ... User support application, 100 ... Application state recognition system, 110 ... Image acquisition part, 120 ... Pre-processing part, 130 ... Feature extraction Part, 140 ... identification part, 150 ... identification dictionary.

Claims (5)

In an application state recognition apparatus for recognizing an operation state of an application operating on a computer,
Image data acquisition means of the application state recognition device acquires the information output by the application to the display device as image data;
Preprocessing means of the application state recognition device performs preprocessing including processing for replacing the variable region of the image data with a predetermined fixed value or fixed pattern for the acquired image data;
An application state feature amount extracting unit extracting a feature amount of the preprocessed image data; and
The application state identification unit stores learning data for identifying the operation state of the application from the feature amount of the image data stored in the storage unit in advance as an identification dictionary, and the operation of the application based on the feature amount of the image data. An application state recognition method comprising: a step of identifying a state. In the identification dictionary, learning data is stored for both the feature amount of the image data in which the variable region of the image data is replaced with a predetermined fixed value or fixed pattern and the feature amount of the image data that is not replaced. The application state recognition method according to claim 1 . The learning data with replacement includes coordinate information of a variable region,
The preprocessing step is characterized in that the coordinate information of the variable area is obtained by referring to the learning data of the identification dictionary, and the area indicated by the coordinate information is replaced with a predetermined fixed value or fixed pattern. 2. The application state recognition method according to 2. In an application state recognition device for recognizing an operation state of an application operating on a computer,
Image data acquisition means for acquiring information output to the display device by the application as image data;
Preprocessing means for performing preprocessing on the acquired image data, including processing for replacing a variable region of the image data with a predetermined fixed value or fixed pattern ;
Feature amount extraction means for extracting the feature amount of the preprocessed image data;
Storage means for preliminarily storing learning data for identifying the operation state of the application from the feature amount of the image data as an identification dictionary;
An application state recognition apparatus comprising: an identification unit that identifies an operation state of an application from the identification dictionary based on the extracted feature amount. An application state recognition program for causing a computer to function as each means of the application state recognition device according to claim 4.

Images