JP6264812B2 - Operation support program, communication terminal, and processing device - Google Patents

Description

  The present invention relates to an operation support program, a communication terminal, and a processing device.

  For example, Patent Document 1 below proposes a technique for remotely operating a multifunction peripheral using a portable terminal having a touch panel superimposed on an LCD. Specifically, an image obtained by photographing the operation unit of the multifunction device is displayed on the LCD, and the multifunction device is caused to execute processing according to the operation of the touch panel. That is, of the images displayed on the LCD, for example, when a key image to which copy processing is assigned is designated, the portable terminal instructs the multifunction device to execute the copy processing.

JP2013-8183A

  In the case of the above technology, it is necessary to create a function for each multifunction device to instruct processing to the multifunction device according to the input operation to the touch panel of the mobile terminal, or when the multifunction device is upgraded, There was a problem that it was necessary to make additional changes.

  The present invention has been made to solve the above-described problems, and an object thereof is a display unit having a display area, a point on the display area, and a point designated by an input medium. A function that is executed by a communication terminal including an input unit that detects an indication point, a communication unit that communicates with the processing device, and a control unit, and that operates the processing device based on an input to the input unit has versatility. It is to provide an operation support program, a communication terminal, and a processing device that can be executed.

In order to achieve this object, the present invention includes a display unit having a display area, an input unit that detects a point on the display area and a point indicated by the input medium, and a process A process executed by the processing device in an operation support program executed by a communication terminal including a communication unit communicating with the device and a control unit and operating the processing device based on an input to the input unit. Image receiving means for receiving, from the processing device , image data representing the image and specific information for specifying the image data for an image that guides an input operation using the input medium for instructing; Display control means for displaying an image represented by image data in the display area; and the input in a state in which the image is displayed by the display control means. The coordinate information of the designated point on the display area detected by the parts, and the type of the input event to said input unit, and the specific information about the image being displayed on said display area to identify the image data representing the image An error that receives from the processing device information indicating that the specific information transmitted by the input information transmitting unit is different from the specific information being set in the processing device. When the receiving means and the error receiving means receive information indicating that the specific information being set is different, the image data specified by the specific information being set in the processing apparatus to the processing apparatus And the image requesting means for requesting the image to be executed.

In the operation support program according to the first aspect, the image receiving unit receives the image data from the processing device, and the display control unit displays the image represented by the image data in the display area. According to this image, it is possible to guide the input operation to the input unit of the communication terminal. Then, the coordinate information of the indication point detected by the input unit is transmitted to the processing device by the input information transmitting unit together with the type of the input event. Here, since the image data is transmitted from the processing device, the processing device can grasp in advance the correspondence between the position of the designated point in the image and the processing to be executed. In this case, the processing device can grasp what processing should be executed by grasping the position of the designated point in the image from the received coordinate information. For this reason, on the communication terminal side, it is not necessary to know what processing of the processing device is instructed by the input operation on the input unit. Therefore, versatility can be given to the function of operating the processing device based on the input to the input unit.
Further, when receiving the image data, the image receiving means receives specific information for specifying the image data. And the specific information about the image currently displayed on the display area is transmitted to the processing device by the input information transmitting means. Then, the processing device collates the specific information of the image being set (the image being set as an image for guiding operations that can be accepted by the processing device) with the specific information of the image being displayed on the display unit of the communication terminal. Thus, it is possible to determine whether the image being set on the processing device side and the image displayed in the display area of the display unit of the communication terminal are not shifted. Therefore, it is possible to cope with a case where a deviation occurs between the image displayed in the display area of the display unit of the communication terminal and the image being set in the processing apparatus.
Further, when the image being displayed on the communication terminal is different from the image being set in the processing device, information to that effect is received by the error receiving means. In this case, the image request means requests the processing device for the image data specified by the specific information being set in the processing device. As a result, the image displayed on the communication terminal can be updated to that being set in the processing device.

In the operation support program according to the second aspect , when the image being displayed on the communication terminal is different from the image being set in the processing apparatus, information to that effect is received by the error receiving means. In this case, by displaying that fact by the error display means, in addition to the effect of the operation support program according to claim 1, there is an effect that it is possible to notify the user of the situation where the error has occurred.

In the operation support program according to claim 3 , in a period from when image data is requested by the image request means to when image data transmitted from the processing device is received and an image represented by the image data is displayed, Note that the image being set in the processing device is different from the image displayed on the communication terminal. When an input operation is performed on the input unit even though the images are different, if the coordinate information or the like is transmitted to the processing device accordingly, the instruction recognized by the processing device is different from that intended by the user. There is a risk. Therefore, in addition to the effect of the operation support program according to any one of claims 1 and 2 , a process is performed by not performing a transmission process related to an input operation performed at least until image data is received. There is an effect that it is possible to suitably suppress occurrence of a situation in which the apparatus erroneously recognizes an instruction that is different from the instruction intended by the user.

In the operation support program according to the fourth aspect , the image data is requested by the image requesting means, whereby the image data is transmitted from the processing device and received by the image receiving means. In this case, there is a problem how the communication terminal acquires the image data when there is a situation in which the image data should be transmitted in the processing device without any event requesting the image data in the communication terminal. In this respect, in addition to the effect of the operation support program according to any one of claims 1 to 3 , by requesting the image data again when an instruction for requesting the image data again is received by the instruction receiving unit. In the case where the image data should be transmitted to the communication terminal on the processing device side, there is an effect that the communication terminal can appropriately receive the image data.

In the operation support program according to the fifth aspect, when receiving an instruction to request image data again, the inquiry unit inquires whether or not to permit the request to be issued again to the processing device. Then, by requesting the image data again when the inquiry result is permission, the image data can be received while reflecting the user's intention in addition to the effect of the operation support program according to claim 4 .

In the operation support program according to claim 6, since the object received by the image receiving means is image data representing an image displayed in the display area of the apparatus-side display unit, any one of claims 1 to 5. In addition to the effect of the operation support program described in (1), the input operation to the terminal side input unit can be performed with a sense similar to the input operation to the device side input unit.

In the communication terminal according to the seventh aspect, an effect similar to the effect of the operation support program according to any one of the first to sixth aspects can be achieved.

In the processing device according to the eighth aspect, image data representing an image for guiding an input operation using the input medium of the communication terminal is transmitted to the communication terminal by the image transmission means. Thereafter, the coordinate information of the designated point specified in accordance with the input operation on the input unit and the input event are received by the input information receiving means. Here, since the image data has been transmitted from the processing device, the processing device should know in advance the correspondence between the position of the indicated point and the type of input event in the image and the processing to be executed. Can do. In this case, it is possible to identify what processing should be executed by grasping the position of the designated point in the image from the received coordinate information. For this reason, the execution means can execute the process specified by the coordinate information and the type of the input event. As described above, in order to execute the process by the execution unit, the communication terminal only needs to transmit the coordinate information and the input event, so that it is not necessary to specify the process to be executed by the execution unit. For this reason, the versatility can be given to the function which operates a processing apparatus based on the input with respect to an input part.
Specific information for specifying image data to be transmitted to the communication terminal is transmitted by the specific information providing means. On the other hand, when coordinate information or the like is transmitted from the communication terminal, the specific information regarding the image being displayed on the communication terminal side is acquired by the specific information acquisition unit. Then, the matching unit determines whether the specific information being set in the processing device matches the acquired specific information. Here, if they do not match, it is considered that the image being set in the processing device (the image being set as an image for guiding the operation that can be accepted by the processing device) does not match the image being displayed on the communication terminal side. For this reason, when the process according to the input event is executed by the execution unit, a process different from the user's intention may be executed by the execution unit. In this regard, since the processing according to the input event is executed by the execution means on condition that they match, it is possible to avoid a situation in which processing different from the user's intention is executed by the execution means.
In addition, when it is determined that there is a mismatch by the collating unit, the fact is transmitted to the communication terminal by the error transmitting unit, so that it is possible to deal with a situation where the communication terminal does not match.
Furthermore, on condition that the request receiving means receives the transmission request, the image data representing the image being displayed on the display unit is transmitted to the communication terminal. For this reason, when a communication terminal requires image data, there exists an effect that it becomes possible to transmit image data. In addition, since image data representing an image displayed in the display area of the display unit is transmitted to the communication terminal, the input operation on the terminal side input unit is performed in a manner similar to the input operation on the device side input unit. Can do.

In the processing apparatus according to claim 9, wherein, in a state where the image data which is not transmitted to the communication terminal is image representing displayed in the display area of the device-side display unit, the image displayed on the display area of the table radical 113 On the condition that it changes, image data representing the image before the change is stored in the storage unit by the storage control means. Thus, in addition to the effect of the treatment apparatus according to claim 8, when the image data is requested by the request receiving means, although different from the image data of the image currently displayed in the display area of the table radical 113, There is an effect that it is possible to transmit image data that represents a previously displayed image and has not yet been transmitted to the communication terminal to the communication terminal.

In the processing apparatus according to claim 1 0, wherein, when the transmission request of the image data is received by the request receiving means when the image data in the storage unit is stored, transmits the image data stored in the storage unit. For this reason, in addition to the effect of the processing device according to claim 9 , the image data that represents the previously displayed image and has not yet been transmitted to the communication terminal is quickly transmitted to the communication terminal. There is an effect that can be.

In the processing apparatus according to claim 1 1, wherein, since the image data representing the image displayed on the display area of the device-side display unit is transmitted to the communication terminal, the input operation on the terminal side input unit, to the device-side input unit There is an effect that it can be performed with a sense according to the input operation.

1 is a system configuration diagram according to a first embodiment. FIG. The figure which illustrates the change process of a screen. The figure which shows an image data map. The flowchart which shows the procedure of the process by the side of a portable terminal. The figure which shows the process at the time of starting an application in a portable terminal. 6 is a flowchart showing a procedure of image enlargement / reduction processing. The figure which shows the production | generation process of coordinate information. The flowchart which shows the procedure of an event reaction process. 6 is a flowchart showing a processing procedure on the MFP side. The flowchart which shows the procedure of the event reaction process concerning 2nd Embodiment.

<First Embodiment>
Hereinafter, a first embodiment according to the present invention will be described with reference to the drawings.

  FIG. 1 shows a system configuration according to the present embodiment. The illustrated multifunction peripheral device (MFP 10) has a printing function, a scanner function, a facsimile function, and the like. The MFP 10 includes a CPU 12, a flash memory 14, a RAM 16, a communication unit 18, an LCD 20, a touch panel 22, a scanner 24, a printer 26, and a modem 28, which are connected to each other via an input / output port 30. The CPU 12 controls each function of the MFP 10 and each unit connected to the input / output port 30 according to fixed values and programs stored in the flash memory 14 and data stored in the RAM 16.

  The flash memory 14 is a rewritable nonvolatile memory, and the RAM 16 is a rewritable volatile memory. The LCD 20 is a liquid crystal display device and displays various screens. A touch panel 22 is provided on the screen of the LCD 20 in an overlapping manner. The touch panel 22 is a user interface that outputs indication point information indicating a position on the touch panel 22 where an input medium such as a finger or a stick is touched or approached.

  Hereinafter, contact or approach to the touch panel 22 by the input medium is referred to as pressing of the touch panel 22 by the input medium, and an increase in the distance between the input medium and the touch panel 22 after pressing is referred to as lifting. Displacement of the pointing point, which is a point on the touch panel 22 that is touched or approached, by displacing the input medium without being lifted after being pressed is referred to as a drag operation. Note that the point on the touch panel 22 in this embodiment is the center point of the area on the touch panel 22 where the contact or approach of the input medium is detected, but is not limited thereto, and the contact or approach of the input medium is detected. An arbitrary point in the region on the touch panel 22 may be used, or the region itself may be read as a point in the present embodiment. These definitions also apply to the touch panel 46 of the mobile terminal 40.

  The scanner 24 reads a document and converts it into image data. The printer 26 prints an image based on the image data on a recording sheet. The modem 28 modulates image data to be transmitted at the time of FAX transmission into a signal that can be transmitted to a telephone line network (not shown), or transmits a signal input from a telephone line network (not shown) as image data. Demodulate. The communication unit 18 is an interface for performing wireless communication. In the present embodiment, the wireless communication performed by the communication unit 18 is Wi-Fi communication (Wi-Fi is a registered trademark) compliant with the IEEE802.11b / g standard.

  The mobile terminal 40 is, for example, a smartphone or a mobile phone. The portable terminal 40 includes a communication unit 42, an LCD 44, a touch panel 46, a voice input / output unit 48, a telephone network communication unit 50, a CPU 52, a flash memory 54, and a RAM 56, which are connected to each other via an input / output port 58. . The CPU 52 controls each unit connected to the input / output port 58 according to a fixed value, a program, or the like stored in the flash memory 54 or the like.

  The flash memory 54 is a rewritable nonvolatile memory, and the RAM 56 is a rewritable volatile memory. The LCD 44 is a liquid crystal display device and displays various screens. The touch panel 46 is a user interface that is provided so as to overlap the LCD 44 and outputs instruction point information indicating a position on the touch panel 46 where an input medium such as a finger or a stick is touched or approached.

  The voice input / output unit 48 is a voice input / output device including a microphone, a speaker, and the like. The telephone network communication unit 50 is a circuit for performing a call via a mobile phone network (not shown). The communication unit 42 is an interface for performing wireless communication. In the present embodiment, the wireless communication performed by the communication unit 42 is Wi-Fi communication (Wi-Fi is a registered trademark) compliant with the IEEE802.11b / g standard. The mobile terminal 40 is Wi-Fi connected to the MFP 10 having the communication unit 18 by an infrastructure mode via the communication unit 42 and an access point (not shown) as a relay device.

  The MFP 10 displays an image for guiding an input operation using an input medium on the display area 20a of the LCD 20, and performs facsimile transmission processing in response to an input operation on the touch panel 22, such as pressing, dragging, and lifting. In addition, various processes such as a printing process and a scanning process are executed. It is difficult to display all of the images including the object that guides the input operation for instructing the execution of these processes on the display area 20a of the LCD 20 at a time because of the size of the display area 20a. For this reason, in this embodiment, it is possible to switch the image displayed on the display area 20a of the LCD 20. 2A is an example of an image displayed on the display area 20a of the LCD 20, and FIG. 2C is another example of an image displayed on the display area 20a of the LCD 20. In FIG. 2A, an object for specifying a facsimile function (a button described as FAX), an object for specifying a copy function (button described as Copy), and an object for specifying a scan function (described as Scan). Button) is displayed. On the other hand, FIG. 2C shows an example in which an object for instructing execution of various processes related to the facsimile function, such as an object for instructing execution of the facsimile function (a button described as FAX Start), is shown.

  In the present embodiment, when the image displayed on the display area 20a of the LCD 20 is changed in the MFP 10, the display (at least the position of the object in the image displayed on the display area 20a is displaced at a speed that can be sensed by the user). Change the image using animation display. That is, when changing from the image before the change shown in FIG. 2A to the image after the change shown in FIG. 2C, the image is changed from the right side of the image before the change (entire screen) with the change of the image. Slide the image (entire screen) gradually. FIG. 2B shows a stage in the middle of sliding. During this halfway period, an input operation is not accepted on the touch panel 22 of the MFP 10.

  The change of the image in the display area 20a of the LCD 20 is basically performed in accordance with an input operation by the user. Specifically, the touch panel 22 is moved up after an object that causes screen transition is pressed. This is because, in the present embodiment, the pressing operation on the touch panel 22 means an operation for selecting the position of the designated point, and the lifting operation means an operation for confirming the position of the designated point. Note that the position of the pointing point that has been pressed is stored in the RAM 16.

  However, in the present embodiment, not only the lifting operation on the touch panel 22, but when the information to be notified to the user is generated inside the MFP 10, the object that causes the screen transition is pressed and not released. In addition, the screen of the display area 20a of the LCD 20 may be changed. Here, the information to be notified to the user includes, for example, a case where the ink remaining amount of the printer 26 has decreased. This screen transition is basically a pop-up display. Note that the pop-up display may be deleted from the display area 20a by being performed for a predetermined time.

  In this embodiment, each screen displayed in the display area 20a of the LCD 20 is given information (screen ID) for specifying the screen. As shown in FIG. 3, the screen ID is associated with an image by an image data map stored in the flash memory 14 of the MFP 10. The image data map is data in which a screen ID and information on necessity of storage are associated for each image data. Here, the information on necessity of storage will be described in detail in the description of the processing (S120) of FIG.

  In addition to the input operation on the touch panel 22, the MFP 10 can also execute various functions in response to instructions from the mobile terminal 40. In order to make this possible, in the present embodiment, an image for guiding an input operation using the input medium displayed on the display area 20a of the LCD 20 of the MFP 10 can be displayed also on the LCD 44 of the portable terminal 40. That is, when the MFP 10 transmits image data representing this image to the portable terminal 40, the portable terminal 40 displays the image represented by the received image data in the display area 44 a of the LCD 44. Thus, the user can perform an input operation using the input medium on the touch panel 46 in accordance with the image displayed in the display area 44a of the LCD 44. Then, the portable terminal 40 outputs the coordinate information of the indicated point on the display area 44a of the LCD 44 accompanying the input operation on the touch panel 46 to the MFP 10. Based on this coordinate information, as will be described in detail later with reference to FIG. 7, the MFP 10 can grasp which position of the image displayed on the display area 20 a of the LCD 20 is designated. Thus, for example, when the indication point is located in the object for instructing execution of the printing process in the display area 44a of the LCD 44, the MFP 10 recognizes that the execution of the printing process is instructed based on the coordinate information. Execute print processing. Thus, the MFP 10 can be operated by an input operation on the touch panel 46 of the portable terminal 40. At this time, the portable terminal 40 only notifies the MFP 10 of the coordinate information of the indication point and the type of input event (pressed, dragged, lifted) to the touch panel 46. In the portable terminal 40, the input to the touch panel 46 is performed. There is no need to know what processing execution instruction is for the MFP 10.

  Hereinafter, processing related to the operation of the MFP 10 by the mobile terminal 40 will be described in detail with reference to the drawings.

  FIG. 4 shows a procedure of processing executed by the mobile terminal 40. This process is realized by the CPU 52 executing an operation support program stored in the flash memory 54. Each process of S14 to S54 is executed each time an input operation on the touch panel 46 is received in the process of S14. That is, the processes from S14 to S54 are processes that can be executed in parallel.

  The CPU 52 executes the LCD data acquisition process when the operation support program (application) is started (S10). That is, as shown in FIG. 5, CPU 52 outputs a signal requesting image data (LCD data) representing an image displayed on display area 20 a of LCD 20 of MFP 10 to MFP 10. Then, the CPU 52 receives the LCD data transmitted from the MFP 10, and displays it in the display area 44 a of the LCD 44.

  Next, the CPU 52 receives additional information transmitted from the MFP 10 (S12). Here, the additional information includes a screen ID for specifying the image requested in S10. The MFP 10 includes the screen ID corresponding to the image data transmitted to the portable terminal 40 in the additional information using the image data map shown in FIG.

  Subsequently, the CPU 52 waits for an input operation on the touch panel 46 (S14). And when CPU52 receives input operation, it is judged whether the input event regarding the input operation is a pressing event (S16). When determining that the event is a pressing event (S16: Yes), the CPU 52 stores the position of the pointing point that has been pressed in the RAM 56, and determines whether or not an input event can be received on the touch panel 46 ( S18). This process is for determining whether or not the process of S46 described later is being performed. If the CPU 52 determines that it is not possible to accept an input event for the touch panel 46 (S18: No), the process proceeds to S14 in order to accept the input event when the input to the touch panel 46 can be accepted. . On the other hand, when the CPU 52 determines that the input event can be received with respect to the touch panel 46 (S18: Yes), the CPU 52 executes an image enlargement / reduction process (S20).

  FIG. 6 shows a procedure for image enlargement / reduction processing. In this process, the pressing event received in S14 is a pressing event that instructs the MFP 10 to execute the process, or one of the continuous pressing events received by performing a pinch-in / pinch-out operation by the input medium. This is for identifying whether the event is a press event. Here, the pinch-in operation is an operation of placing two fingers, for example, a thumb and an index finger on the touch panel and pinching with the two fingers, and is performed when the image is reduced. . The pinch-out operation is an operation of placing two fingers, for example, a thumb and an index finger on the touch panel, and pushing and spreading with the two fingers, for example, when enlarging an image. . Therefore, in the case of a pinch-in and pinch-out operation, it is an input operation that is irrelevant to the MFP 10 because it instructs to reduce or enlarge an image displayed in the display area 44 a of the LCD 44 of the portable terminal 40.

  The CPU 52 determines whether or not the waiting process described in step S61 has been executed (S60). When the CPU 52 determines that the waiting process is not being executed (S60: No), the CPU 52 starts the waiting process for waiting for a predetermined time (S61). Here, the predetermined time is set according to a time difference between the pair of pressing timings when two fingers as input media are pressed on the touch panel 46 so that the user performs a pinch-in operation or a pinch-out operation. Specifically, the time difference is set to a value that can normally be taken. For example, in this embodiment, it is set to 0.5 seconds. The CPU 52 determines whether another pressing event has occurred during the waiting process (S62). This process is for determining whether pinch-in or pinch-out is performed. When the CPU 52 determines that another pressing event has occurred during the waiting process (S62: Yes), the CPU 52 displays it in the display area 44a of the LCD 44 in accordance with the movement state of the pair of instruction points corresponding to the pair of pressing events. Processing for enlarging or reducing the image to be performed is executed (S64). In practice, it is desirable that the process of S62 is not executed after waiting for a predetermined time, but is repeatedly executed until an affirmative determination is made before the predetermined time elapses. Thereby, when an affirmative determination is made in S62, it is possible to quickly shift to the process of S64.

  On the other hand, when the CPU 52 determines that another pressing event does not occur during the waiting process (S62: No), the CPU 52 proceeds to the process of S22 in FIG.

  Thereafter, when the other input medium approaches or comes into contact with the touch panel 22 before the take-off event occurs, the CPU 52 makes an affirmative determination again in the process of S16 of FIG. 4 and proceeds to the image enlargement / reduction process. , It is determined that the waiting process of S61 is being executed (S60: Yes), and the process proceeds to S72. At this point, since there are two indication points, there is a possibility that the user is trying to pinch-in and pinch-out, but the indication points did not become two during the waiting process time. In the process of S72, it is discriminated whether the operation is a pinch-in / pinch-out operation or whether another finger happens to press the touch panel 46 during dragging.

  Specifically, the CPU 52 determines whether or not the drag storage count incremented by the process of S36 described later is equal to or less than a certain number (S72). The drag storage count indicates the sampling count when the drag operation is performed, and the drag storage count increases, indicating that the drag time is longer. Here, the fixed number is set to a value that is difficult to assume in the case of pinch-in and pinch-out. If the CPU 52 determines that the number exceeds a certain number (S72: No), the CPU 52 determines that the time for the drag operation is too long for the pinch-in / pinch-out operation, and proceeds to the process of S22 in FIG. On the other hand, if the CPU 52 determines that the number is less than a certain number (S72: Yes), it is considered that a pinch-in / pinch-out operation is performed, and therefore notifies the MFP 10 that the transmission of the pressing event is canceled (S74). ). This is because, when the CPU 52 makes an affirmative determination in S72, the user intends to perform a pinch-in operation or a pinch-out operation, but another one point is detected after one point is detected on the touch panel 46. Since it took time to be performed, a negative determination is made in the process of S62, and a press event is transmitted to the MFP 10 by the process of S24 through the process of S22 described later. This cancellation process is a premise of the process of S160 of FIG. When completing the process of S74, the CPU 52 executes an enlargement / reduction process according to the movement state of the pair of instruction points (S64).

  The CPU 52 determines whether or not all indication points are detected after the execution of the process of S64. In other words, it is determined whether all the input media have moved away from the touch panel 22 (S66). If it is determined that all indicated points are no longer detected (S66: No), a process of initializing the drag memory count is executed (S68), and the process of S14 of FIG. 4 is waited for a new input to the touch panel 46. Return to. Note that the drag storage count changes to the value after initialization by the processing of S68 is actually a case where the processing of S74 described later is executed. If an affirmative determination is made in S62, the processing of S68 is actually executed. Is not necessary.

  When the process of S20 in FIG. 4 is completed, that is, when a negative determination is made in S62 and S72, the CPU 52 executes a process of converting the pressed position (S22). For example, as shown in FIG. 7A, the coordinates (drawing start coordinates: X, Y) and the coordinates (A, B) of the pressed point in the display area 44a of the LCD 44 are used as a reference for displaying an image. If the difference (A−X, B−Y) is divided by the enlargement factor k of the displayed image, the value {(A−X) / k, (B−Y) / k} is calculated. Good. The difference (A−X, B−Y) is the coordinate of the pressed point when the drawing start coordinates (X, Y) are re-taken from the origin. Here, the enlargement factor k is a ratio of the size of the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 to the size of the image displayed on the display area 20a of the LCD 20 of the MFP 10. Here, the size is not limited to the actual length, and may be, for example, the number of display pixels. In this case, for example, when the area per pixel is smaller in the LCD 44 of the portable terminal 40 than in the LCD 20 of the MFP 10, the actual size of the displayed image is not limited even if the enlargement ratio is 1. What is displayed on the LCD 44 is smaller than what is displayed on the LCD 20.

  Even if the image is enlarged or reduced in the display area 44a of the LCD 44 of the portable terminal 40, the conversion process may be performed by any of the images displayed in the display area 20a of the LCD 20 in the MFP 10. This is intended to make it possible to grasp whether the point is a pressing point. Such an aim can also be realized by the ratio of the length from the reference point to the pressed point with respect to the length of the image in each component of the two-dimensional coordinates, as shown in FIG. 7B. In other words, the pair of end portions on the diagonal line of the image is the drawing start coordinate (X, Y) and the drawing end coordinate (Xe, Ye), and the length of the displayed image and the drawing for each of the two-dimensional coordinate components. The ratio of the length of the pressed point to the start coordinate {(AX) / (Xe-X), (BY) / (Ye-Y)} may be used.

  Subsequently, in the process of S24 of FIG. 4, the CPU 52 determines that the pressed position information obtained by the above process (the coordinate information of the pointing point instructed by pressing), the type of the input event is the pressed event, The screen ID of the image displayed in the display area 44a is transmitted to the MFP 10 (S24). Subsequently, the CPU 52 executes an event reaction process (S26). FIG. 8 shows this process.

  In this series of processing, the CPU 52 notifies that the screen ID of the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 is different from the screen ID of the image displayed on the display area 20a of the LCD 20 of the MFP 10. Is received from the MFP 10 (S80). This notification is made by the process of S140 of FIG. 9 described later, which is executed in the MFP 10 in accordance with the process of S24 of FIG. Here, when a pressing operation is performed on the touch panel 46 of the portable terminal 40 and a pressing event is transmitted to the MFP 10, the image in the display area 20 a of the LCD 20 of the MFP 10 does not change. This is because, as described above, in the present embodiment, the screen is changed by performing an operation for confirming the pointing point (a lifting operation). Nevertheless, the screen ID may differ, as described above, when information to be notified to the user is generated in the MFP 10, it is displayed in the display area 20 a of the LCD 20 regardless of the input operation by the user. This is because the image is changed.

  And CPU52 complete | finishes an event reaction process, when notification of a different effect is not received for the predetermined time (S80: No). On the other hand, when the CPU 52 receives a notification indicating that there is a difference (S80: Yes), the CPU 52 executes LCD data acquisition processing assuming that there is LCD data to be acquired from the MFP 10 (S82). This process is the same as the process of S10 in FIG. Subsequently, the CPU 52 executes additional information acquisition processing (S84). This process is the same as the process of S12 of FIG. 4, but in the process of S84, in addition to the screen ID, next screen presence / absence information is also acquired. Subsequently, the CPU 52 displays an image for notifying that the screen is updated because the screen ID is different in the display area 44a of the LCD 44 (S86). This may be realized by, for example, pop-up display. This process notifies the user that the screen is changed because the image displayed on the display area 20a of the LCD 20 of the MFP 10 and the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 do not match. It is made for that purpose.

  Next, the CPU 52 initializes the number of times of drag storage described later (S88). Then, the CPU 52 determines whether there is a next image (next image) to be acquired from the MFP 10 based on the additional information received by the process of S84 (S90). In this process, the mobile terminal 40 represents each of a plurality of images due to a situation in which the image displayed on the display area 20a of the LCD 20 is changed on the MFP 10 side independently of the mobile terminal 40. When image data is received in order and displayed in the display area 44a of the LCD 44, it is assumed that the additional information includes information indicating that there is a next image. This setting will be described in detail in the processing of S114 to S122 in FIG. When determining that there is no next image (S90: No), the CPU 52 ends the event reaction process. On the other hand, when determining that there is a next image (S90: Yes), the CPU 52 executes a waiting process (S92) and returns to the process of S82. Here, the waiting time of the waiting process is set to a length that allows the user to sufficiently check the image currently displayed in the display area 44 a of the LCD 44.

  When the event reaction process ends, the CPU 52 returns to the process of S14 in FIG. 4 to wait for a new input operation on the touch panel 46.

  When determining that the newly generated event is not a pressing event (S16: No), the CPU 52 determines whether or not the newly generated event is a drag event (S28). If the CPU 52 determines that the event is a drag event by detecting the displacement of the indicated point on the touch panel 46 (S28: Yes), can the input event be accepted for the touch panel 46 as in the process of S18 above? It is determined whether or not (S30). If the CPU 52 determines that it is not possible to accept an input event for the touch panel 46 (S30: No), the CPU 52 returns to S14 to accept this when the input to the touch panel 46 is possible. On the other hand, when the CPU 52 determines that the input event can be received with respect to the touch panel 46 (S30: Yes), the CPU 52 executes a drag position conversion process (S32). In this process, the position of the current indication point is determined by the LCD 20 of the MFP 10 regardless of the size of the image displayed in the display area 44a of the LCD 44 in the manner of the process shown in FIG. 7A or 7B. It is a process which converts into the value (coordinate information) which can be specified as a position on the image displayed on the display area 20a.

  Subsequently, the CPU 52 transmits to the MFP 10 that the type of event that has newly occurred is a drag (drag event) and the coordinate information of the pointing point that accompanies the drag calculated by the process of S32 (S34). Subsequently, the CPU 52 executes a process of storing the number of drag events indicating the number of drag position samplings in the RAM 56 in S32 (S36). This process is a prerequisite for the process of S72 of FIG. Then, when completing the process of S36, the CPU 52 returns to the process of S14 to wait for a new input to the touch panel 46.

  On the other hand, when the CPU 52 determines that the newly generated event is not a drag event (S28: No), the CPU 52 determines whether the newly generated event is a lift event (S38). If the CPU 52 determines that the event is not a lift event (S38: No), the CPU 52 returns to the process of S14 in order to wait for a new input to the touch panel 46. On the other hand, when determining that the event is a lift event (S38: Yes), the CPU 52 determines whether or not an input event can be received with respect to the touch panel 46, similarly to the process of S18 (S40). . If the CPU 52 determines that it is not possible to accept an input event for the touch panel 46 (S40: No), the CPU 52 returns to the process of S14 to accept this when the input to the touch panel 46 is acceptable. . On the other hand, when the CPU 52 determines that the input event can be received with respect to the touch panel 46 (S40: Yes), the CPU 52 executes a lift position conversion process (S42). In this process, the position where the pointing point disappears, in other words, the position of the last detected pointing point (pressed position or drag position) is determined in the manner shown in FIG. 7A or 7B. This is a process of converting to a value that can be specified as a position on the image displayed on the display area 20a of the LCD 20 of the MFP 10 regardless of the size of the image displayed on the display area 44a of the LCD 44.

  Subsequently, the CPU 52 indicates that the newly generated event type is a lift, the screen ID of the image displayed on the LCD 44, and the lift position information calculated in the process of S42 (disappears with the lift). The coordinate information of the designated point) is transmitted to the MFP 10 (S44). Subsequently, the CPU 52 starts a process for prohibiting event reception (S46), and executes a waiting process (S48). Then, when the waiting process is completed, the CPU 52 executes an LCD data acquisition process that is the same process as the process of S10 (S50), and executes an additional information acquisition process that is the same process as the process of S12 (S52). ). In the process of S52, the next screen presence / absence information is acquired in addition to the screen ID. The waiting process is for waiting for transmission of a request signal for LCD data for a time required for changing the image displayed on the display area 20a of the LCD 20 of the MFP 10 due to the take-off event. As described above with reference to FIG. 2, this is based on the premise that an animation is displayed when an image displayed in the display area 20a of the LCD 20 is changed.

  If LCD data is requested from the portable terminal 40 to the MFP 10 during the animation display period, the image displayed on the MFP 10 may not match the image displayed on the portable terminal 40. In such a case, if an input operation is accepted, the MFP 10 may malfunction. That is, during the period in which the LCD data being animated is transmitted to the mobile terminal 40 and the image represented by the data is displayed in the display area 44a of the LCD 44, the image displayed on the display area 20a of the LCD 20 of the MFP 10 is changed. When completed, the MFP 10 can already perform an input operation on the touch panel 22. At this time, on the portable terminal 40 side, an input operation is performed on the touch panel 46 based on the image shown in FIG. 2B displayed in the display area 44a of the LCD 44, and the coordinate information of the indicated point is transmitted to the MFP 10. Then, the MFP 10 may erroneously recognize that an input operation has been performed on a location different from the location intended by the user in the image displayed on the display area 20a of the LCD 20. If the misrecognized indication point is located in the object indicating the location where the indication point should be in order to instruct the processing executed by the MFP 10 in the display area 20a of the LCD 20, the MFP 10 may malfunction. . Therefore, the processing of S46 and S54 in FIG. 4 prevents the input operation to the touch panel 46 of the portable terminal 40 during the animation display period on the MFP 10 side.

  When completing the process of S52, the CPU 52 cancels the event reception prohibition process (S54), and returns to the process of S14 to wait for a new input to the touch panel 46. Incidentally, it is prohibited to execute this process at the end of the process of S48 on the condition that the image displayed in the display area 44a of the LCD 44 is changed to the image displayed in the display area 20a of the LCD 20. This is for canceling the processing.

  FIG. 9 shows a processing procedure on the MFP 10 side. This process is executed by the CPU 12 of the MFP 10.

  The CPU 12 executes standby processing such as the type of event and coordinate information transmitted from the mobile terminal 40 (S100). When an event occurs, the CPU 12 determines whether the event is a request for LCD data (S102). If the CPU 12 determines that an event for requesting LCD data has occurred (S102: Yes), the CPU 12 determines whether there is stored LCD data (S104). In this process, as described above, the image displayed on the display area 20a of the LCD 20 of the MFP 10 generates information to be notified to the user in the MFP 10 regardless of the input operation to the touch panel 46 of the portable terminal 40. It is provided in view of the fact that it may be changed. That is, in this case, image data that has not yet been transmitted to the portable terminal 40 is stored in the RAM 16 by the process of S122 described later. Therefore, in the process of S104, an image representing the image being displayed on the display area 20a of the LCD 20 In addition to the data, it is determined whether there is image data stored in the RAM 16. When determining that the stored LCD data does not exist (S104: No), the CPU 12 acquires a screen ID that is an ID of the image data displayed in the display area 20a of the LCD 20 (S106). This process can be executed based on the image data map shown in FIG. Subsequently, the CPU 12 stores the acquired screen ID in the RAM 16 as a transfer screen ID to be transferred to the portable terminal 40 (S108). Then, the CPU 12 transfers the LCD data and the transfer screen ID as additional information to the portable terminal 40 (S110). Note that the additional information transfer process is actually executed in response to a request for additional information from the portable terminal 40 in addition to the LCD data request by the processes of S12, S52, and S84 in FIG. The

  On the other hand, when determining that the event is not an LCD data acquisition instruction event (S102: No), the CPU 12 determines whether or not an event that causes a transition of the screen of the LCD 20 has occurred (S116). This process is for determining whether or not an event has occurred in which the screen of the LCD 20 changes despite no instruction from the portable terminal 40. The screen transition in this case is a pop-up display or the like made when information to be notified to the user is generated in the MFP 10 as described above. Here, in the image data map of FIG. 3, different screen IDs are assigned when the pop-up display is performed and when the pop-up display is not performed.

  When determining that a screen transition event has occurred (S116: Yes), the CPU 12 determines whether or not the transition source screen ID matches the transfer screen ID stored in S108 (S118). This process is for determining whether the image before the transition has not yet been transferred to the portable terminal 40. When determining that they do not match (S118: No), the CPU 12 determines whether the screen needs to be stored (S120). This process is executed by referring to the image data map shown in FIG. That is, whether or not storage is necessary is determined in advance in the image data map, and based on this, it is determined whether or not storage is necessary. And CPU12 preserve | saves LCD data in RAM16, when a memory | storage is required (S120: Yes) (S122). Then, the CPU 12 saves the LCD data, determines that they match in S118 (S118: Yes), and further determines that the screen does not need to be stored (S120: No), the process of changing the screen is performed. Execute (S124) and shift the process to S100 in order to wait for a new event to occur.

  If the CPU 12 determines that the LCD data stored in the process of S104 is present (S104: Yes), the CPU 12 acquires the stored LCD data (S112). Next, the CPU 12 transfers the LCD data and the additional information (S114), and returns to the process of S100 to wait for a new event. Here, the additional information includes information indicating that there is a next image. This is a setting for causing the portable terminal 40 to transmit a signal requesting LCD data again. That is, in the present embodiment, when the LCD data request is issued from the mobile terminal 40, the setting is such that the LCD data is transferred from the MFP 10 to the mobile terminal 40, and the MFP 10 transmits the request to the mobile terminal 40 without receiving the request. Does not voluntarily transmit LCD data. Here, the situation where there is stored LCD data is a situation where a negative determination is made in S118 and an affirmative determination is made in S120, which is a situation that cannot be grasped on the portable terminal 40 side. For this reason, information indicating that there is a next image is added as additional information. Accordingly, when the CPU 52 makes an affirmative determination in S90 of FIG. 8, the CPU 52 requests LCD data again by the processing of S82. Note that the additional information transfer process in the process of S110 actually requires additional information from the portable terminal 40 separately from the LCD data request by the processes of S12, S52, and S84 in FIG. Will be executed accordingly.

  When determining that the newly generated event is neither a request for LCD data nor a transition of the main body screen (S116: No), the CPU 12 determines whether or not it is a pressing event (S126). If the CPU 12 determines that the event is a pressing event (S126: Yes), the CPU 12 executes a process of acquiring (receiving) the screen ID added to the event (S128). This process is based on the process of S24 in FIG. Subsequently, the CPU 12 determines whether or not the acquired screen ID matches the screen ID of the image displayed in the display area 20a of the LCD 20 (S132). This process is for determining whether or not the image displayed in the display area 20a of the LCD 20 of the MFP 10 matches the image displayed in the display area 44a of the LCD 44 of the portable terminal 40. If the CPU 12 determines that they match (S132: Yes), the CPU 12 executes the processes of S134 to S138, assuming that the pressed event notified from the portable terminal 40 is valid as an instruction to the MFP 10. That is, the CPU 12 first performs a process of converting the pressed position data if necessary (S134).

  Subsequently, the CPU 12 stores the pressed position in the RAM 16 (S136). This process is a premise of the process of S164 described later. Then, the CPU 12 executes a process accompanying the pressing (pressing process) (S138), and returns to the process of S100 in order to wait for a newly occurring event. This processing is, for example, processing for visually displaying that an object is selected when an object is present at the pressed position. However, the display area 20a of the LCD 20 may not be changed at all and may be processing only within the MFP 10.

  In contrast, if the CPU 12 determines that the screen IDs do not match in the process of S132 (S132: No), the CPU 12 notifies the mobile terminal 40 that the screen IDs do not match (S140), and waits for a new event to occur. Return to the process. The process of S140 is a premise of the process of S80 of FIG. In this case, the pressing process (S138) is not executed. This is because when the image displayed on the display area 20a of the LCD 20 of the MFP 10 and the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 do not match, the pressing event notified from the portable terminal 40 is the MFP 10. This is because it is considered that it is not effective as an instruction for processing.

  When determining that the newly generated event is not a pressing event (S126: No), the CPU 12 determines whether the event is a drag instruction event (S142). If the CPU 12 determines that the event is a drag instruction event (S142: Yes), the CPU 12 executes a process of converting the drag position data as necessary in the process of S134 (S144). Subsequently, the CPU 12 determines whether or not a display corresponding to the drag, for example, a display for moving the dragged image, needs to be displayed on the LCD 20 of the MFP 10 in accordance with the drag operation on the touch panel 46 of the portable terminal 40 (S146). . This process can be executed by storing a value that determines whether or not it is necessary in advance in the flash memory 54 of the MFP 10. Whether or not the display according to the drag is necessary may be determined at the time of shipment of the MFP 10 or may be selectable by the user. If the CPU 12 determines that an image being dragged is necessary (S146: Yes), the CPU 12 displays a display corresponding to the drag on the LCD 20 of the MFP 10 in accordance with the drag operation on the touch panel 46 of the portable terminal 40 (S148). On the other hand, if the CPU 12 determines that the image being dragged is not necessary (S146: No), the CPU 12 executes processing for saving the current indication point accompanying the dragging in the RAM 16 (S150), and waits for a new event to occur. Therefore, the process returns to S100.

  When determining that the newly generated event is not a drag instruction event (S142: No), the CPU 12 determines whether the event is a lift instruction event (S152). If the CPU 12 determines that the event is a lift instruction event (S152: Yes), the CPU 12 determines whether or not the position of the instruction point that has changed from the time of pressing is saved by the process of S150 (S154). . When the CPU 12 determines that the position of the changed indication point is stored (S154: Yes), the CPU 12 executes a process of displacing the display of the indication point in the display area 20a of the LCD 20 to the stored position ( S156). Then, when this process is completed or when it is determined that the position of the changed indication point is not stored (S154: No), the CPU 12 executes a lift-off process (S158) and waits for a newly generated event. Therefore, the process returns to S100. The lifting process is a process executed in the MFP 10 in accordance with the lifting operation on the touch panel 46 of the portable terminal 40. For example, if the selection of the object is confirmed by performing a lifting operation on the object displayed on the display area 44a of the LCD 44 of the portable terminal 40, a visual display corresponding to this is displayed. It may be executed on 20a.

  When the CPU 12 determines that the newly generated event is not a take-off instruction event (S152: No), the CPU 12 determines whether the event is a press cancel event by the process of S74 of FIG. 6 (S160). If the CPU 12 determines that the event is not a press cancel event (S160: No), the CPU 12 returns to the process of S100 to wait for a newly generated event. On the other hand, when determining that the event is a press cancel event (S160: Yes), the CPU 12 determines whether a drag event has occurred after the press operation (S162). If it is determined that a drag event has occurred (S162: Yes), a process of returning the display position of the indicated point on the LCD 20 and the position of the object displaced along with this to the pressed position is executed (S164). This is for returning the display position to the pressed position first when the pressed event is canceled after the display position of the object selected by the pressing is changed by the drag operation continuing after the pressed operation. When this process is completed or when it is determined that there is no drag event (S162: No), a press cancel process is executed (S166), and the process returns to S100 to wait for a new event to occur. Here, for example, when the display indicating that the pressed object is selected, the pressing cancellation process means stopping the visual display.

  According to the embodiment described above, the following effects can be obtained.

  (1) The image displayed on the display area 20a of the LCD 20 of the MFP 10 is displayed on the display area 44a of the LCD 44 of the portable terminal 40. As a result, the input operation on the touch panel 46 of the portable terminal 40 can be performed in a manner similar to the input operation on the touch panel 22 of the MFP 10.

  (2) The coordinate information ((x, y) in FIG. 7 (a) or (m, n) in FIG. 7 (b)) determined according to the input operation on the touch panel 46 of the portable terminal 40 is converted into the MFP 10 (S24, S34, S44 in FIG. 4). Accordingly, since it is not necessary for the portable terminal 40 to know what processing of the MFP 10 is to be performed by the input operation on the touch panel 46, the function of operating the MFP 10 based on the input on the touch panel 46 is provided. It can have versatility.

  (3) The screen ID is associated with the image data (FIG. 3), the screen ID is received when the portable terminal 40 receives the LCD data, and the screen ID is transmitted when the coordinate information or the like is transmitted to the MFP 10. . As a result, the MFP 10 can grasp whether or not the image displayed in the display area 44 a of the LCD 44 of the portable terminal 40 matches the image displayed in the display area 20 a of the LCD 20 of the MFP 10. Therefore, the MFP 10 can deal with a mismatch when it occurs.

  (4) When it is determined that the image displayed in the display area 44a of the LCD 44 of the portable terminal 40 and the image displayed in the display area 20a of the LCD 20 of the MFP 10 do not match (S132: No) The terminal 40 side requested LCD data (S82). Thereby, the situation which is not in agreement can be eliminated.

  (5) When it is determined that the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 and the image displayed on the display area 20a of the LCD 20 of the MFP 10 do not match (S132: No), the MFP 10 The LCD data is received and the display is updated, and the fact that the image is updated due to the mismatch is displayed (S86). Thereby, the user can be made aware of the situation.

  (6) Additional information including a screen ID and LCD data are requested and received separately. As a result, the LCD data (for example, PDF data) transmitted from the MFP 10 can be converted (for example, converted to JPEG) in a state where no other data is attached. Compared with the case where data is combined, the conversion process can be performed easily.

  (7) An input operation on the touch panel 46 is not accepted until the LCD data is requested and received and displayed in the display area 44a of the LCD 44 (the process of S10 in FIG. 4). (The point which provided S14 later, the point which transfers to the process of S14 with the completion | finish of the process of S46-S54, S26). For this reason, the input operation of the touch panel 46 is not accepted in a period in which the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 and the image displayed on the display area 20a of the LCD 20 of the MFP 10 are inconsistent. Thus, it is possible to preferably avoid a situation in which the MFP 10 erroneously recognizes an instruction different from the instruction intended by the user.

  (8) Information on the presence or absence of the next image is included in the additional information. Accordingly, the LCD data can be appropriately requested from the portable terminal 40 side in the setting for transmitting the LCD data from the MFP 10 side to the portable terminal 40 side on the condition that there is a request from the portable terminal 40 side.

  (9) The request for transmission of LCD data from the mobile terminal 40 is a condition for the MFP 10 to transmit LCD data to the mobile terminal 40. Thereby, the timing at which the LCD data is transmitted from the MFP 10 to the portable terminal 40 can be set to the timing at which the portable terminal 40 needs the LCD data. Further, even when there are a plurality of portable terminals 40 that communicate with the MFP 10, it is possible to appropriately transmit the LCD data to the portable terminals 40 that require the LCD data.

  (10) An image representing an image before the transition on the condition that the screen transitions in a state where the image data of the image displayed on the display area 20a of the LCD 20 of the MFP 10 has not yet been transmitted to the portable terminal 40 The data was stored in the RAM 16 (S122). Thereby, the image data of the image before the transition can be transmitted to the portable terminal 40.

  (11) When the lifting operation is performed on the touch panel 46 of the portable terminal 40 and the lifting operation is transmitted to the MFP 10 as the type of the input event, transmission of LCD data is requested (S50 in FIG. 4). Here, the lifting operation is an operation for confirming the selection of the position on the image, whereby the image displayed on the display area 20a of the LCD 20 of the MFP 10 tends to change. Therefore, LCD data can be requested at an appropriate timing.

  (12) Prior to requesting the transmission of the LCD data in response to the lifting operation on the touch panel 46 of the portable terminal 40, the mobile terminal 40 waits for a predetermined period (S48 in FIG. 4). Thus, it is possible to avoid a situation in which the LCD data is transmitted to the portable terminal 40 in response to the LCD data transmission request from the portable terminal 40 before the image replacement is completed on the MFP 10 side.

  (13) The coordinate information ((x, y) in FIG. 7A, (m, n) in FIG. 7B) transmitted from the portable terminal 40 to the MFP 10 is displayed in the display area 44a of the LCD 44. Regardless of whether the image is enlarged or reduced, normalization information for specifying the position of the pointing point when the image has a specified size is included. Therefore, regardless of whether the image displayed in the display area 44a of the LCD 44 is enlarged or reduced, in the MFP 10, the input operation on the touch panel 46 of the portable terminal 40 instructs execution of any process. Can be grasped.

  (14) The coordinate information transmitted from the portable terminal 40 to the MFP 10 is information integrated with the standardized information. Thereby, the coordinate information transmitted from the portable terminal 40 to the MFP 10 can be made simple information.

  (15) When there are a plurality of designated points for the display area 44a of the LCD 44 of the portable terminal 40, the type of input event and coordinate information are not transmitted to the MFP 10 (in the case of Yes in S62 of FIG. 6). Configuration). As a result, a situation in which unnecessary information is transmitted to the MFP 10 can be avoided.

  (16) If another pressing event occurs again after the pressing event is detected and before the lift event occurs, a pressing event cancel event is transferred to the MFP 10 (S74 in FIG. 6). As a result, when the input operation on the touch panel 46 of the portable terminal 40 instructs the enlargement or reduction process, it is possible to avoid a situation where erroneous information about the input event is reflected in the MFP 10.

  (17) The lift event after the cancellation event of the pressing event is transferred (S74 in FIG. 6) is not transferred to the MFP 10 (settings in S66 and S68 in FIG. 6). Thereby, it is possible to avoid a situation where unnecessary input event information is transmitted to the MFP 10.

  (18) When a press cancel event is received, cancel processing is executed (S166 in FIG. 9). Thus, even if the type of the input event transmitted from the MFP 10 is later found to be inappropriate, this can be dealt with.

  (19) When a push cancel event is received and a drag event has occurred after the cancel event to be canceled, processing is performed to return the indicated point to the position when the press event was made (FIG. 9). S164). Thereby, on the LCD 20 of the MFP 10, the object moved by the drag can be returned to the original position. For this reason, the object can be displayed at the position before the pressing event when the pressing cancellation process (S166) is performed.

<Second Embodiment>
Hereinafter, the second embodiment will be described with reference to the drawings with a focus on differences from the first embodiment.

  In the first embodiment, when requesting LCD data from the mobile terminal 40 to the MFP 10, if the mobile terminal 40 receives a notification that there is a next image from the MFP 10, the mobile terminal 40 detects that the input medium approaches or touches the touch panel 46. Even if not, the next image is requested to the MFP 10 to acquire the LCD data, and the image represented by the LCD data is displayed in the display area 44 a of the LCD 44. On the other hand, in the present embodiment, when a notification that there is a next image is received, the user is asked for permission to display the next image before the next image is displayed.

  FIG. 10 shows a procedure of event reaction processing according to the present embodiment. This process is realized by the CPU 52 executing an operation support program stored in the flash memory 54 of the portable terminal 40. Note that, in FIG. 10, the same step numbers are assigned for convenience corresponding to the processing of FIG. 8.

  When determining that there is a next image in the process of S90, the CPU 52 displays an image for inquiring whether the user requests the next image on the LCD 44 (S170). Subsequently, as a result of the inquiry, the CPU 52 determines whether or not the user permits a request for the next image (S172). Then, the CPU 52 waits until the user permits the request for the next image (S172: No), and when the user permits the request for the next image (S172: Yes), the CPU 52 proceeds to the LCD data acquisition process (S82).

  According to the present embodiment described above, the following effects can be obtained in addition to the effects of the first embodiment.

  (20) When acquiring information indicating that there is a next image as additional information when acquiring LCD data, an inquiry is made as to whether or not a re-request is permitted for the LCD data. It was performed again (S172: Yes). Thereby, the acquisition process of LCD data can be performed according to a user's intention.

<Correspondence between each means and the embodiment>
Display unit (terminal side display unit) ... LCD 44, device side display unit ... LCD 20, input unit (terminal side input unit) ... touch panel 46, device side input unit ... touch panel 22, communication unit ... communication unit 42, control unit ... CPU 52, processor ... MFP 10, the storage unit ... RAM 16, the image receiving means ... S10, S12, S50, S52 , S82, S84, the display control unit ... S10, S50, S82, input information transmitting means ... S24, S34, S44, an error received Means S80, error display means S86, image request means S10, S50, S82, instruction receiving means S12, S52, S84, inquiry means S170, image transmission means S110, 114, input information receiving means S100, Execution means ... S138, S148, S158, specific information providing means ... S110, S114, specific information acquisition hand Step S128, collating means S132, error transmitting means S140, request receiving means S102, storage control means S122, specific information screen ID.

<Other embodiments>
The present invention has been described based on the above embodiment, but the present invention is not limited to the above embodiment, and various modifications can be easily made without departing from the spirit of the present invention. It can be guessed. In each embodiment, a part or a plurality of parts of the configuration of the other embodiments are added to the embodiment or replaced with a part or a plurality of parts of the configuration of the embodiment. You may make it comprise and modify. Hereinafter, an embodiment as a modification of the above embodiment will be described.

  In the above embodiment, the waiting process (S48 in FIG. 4) is provided corresponding to the time required for the animation display when the image displayed on the display area 20a of the LCD 20 of the MFP 10 is changed. However, the present invention is not limited to this. For example, when LCD data is requested from the portable terminal 40 to the MFP 10, the LCD data may be transmitted after waiting for the animation display to end in the MFP 10. In this case, it is desirable that the portable terminal 40 does not accept input to the touch panel 46 until the LCD data is received and the image displayed in the display area 44a of the LCD 44 is switched after the LCD data is requested.

  In the above embodiment, when the image displayed on the display area 20a of the LCD 20 of the MFP 10 and the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 do not match, the portable terminal 40 requests the LCD data from the MFP 10. However, it is not limited to this. For example, when it is determined that the MFP 10 does not match, the LCD data may be voluntarily transmitted to the portable terminal 40. In this case, if the screen ID is further transmitted to the portable terminal 40, the portable terminal 40 can grasp that the image data transmitted from the MFP 10 and the image being displayed in the display area 44a of the LCD 44 are different. .

  The timing for transmitting the screen ID from the portable terminal 40 to the MFP 10 is not limited to the time when coordinate information or the like is transmitted in accordance with the pressing event (S24). For example, the screen ID may be transmitted from the portable terminal 40 to the MFP 10 at regular intervals.

  In FIG. 2, when the screen itself is changed, the screen itself is changed in view of the fact that the object indicating the location where the indication point should be located in instructing the processing executed by the MFP 10 is displaced in the display area. At this time, the input operation to the touch panel 46 of the portable terminal 40 is not accepted, but the present invention is not limited to this. For example, when the entire image displayed in the display area 20a of the LCD 20 of the MFP 10 is not changed, but only the object indicating the location where the instruction point should be located in instructing the processing executed by the MFP 10, the object is displaced. An input operation on the touch panel 46 of the mobile terminal 40 may not be accepted.

  A setting for sending LCD data and additional information corresponding to the LCD data from the MFP 10 by requesting the LCD data is not limited to the request for the LCD data and the request for the additional information separately from the portable terminal 40 to the MFP 10. It may be.

  In the processing of S46 and S54 in FIG. 4, the input operation to the touch panel 46 of the portable terminal 40 is not accepted during the animation display period on the MFP 10 side, but this is not restrictive. For example, the display of the display area 44a of the LCD 44 is changed according to the input operation, and the fact that the input operation is invalidated is further displayed on the display area 44a of the LCD 44, and the input event and coordinate information related to the input operation are displayed on the MFP 10 Transmission may not be performed.

  In the process of FIG. 4, the input operation to the touch panel 46 is not accepted during the period when the waiting process of S <b> 48 is performed. However, even during this period, the pinch-out operation for instructing the enlargement of the screen is performed. As described above, an input operation unrelated to the instruction to the MFP 10 may be accepted.

  In the method of FIG. 7B, the length of each coordinate component from the reference point to the predetermined point is set with the reference point as the drawing start coordinate (X, Y) and the predetermined point as the drawing end coordinate (Xe, Ye). The present invention is not limited to the information used as standardized information for specifying the position of the pointing point when the image has a prescribed size. For example, the predetermined point may be set by the coordinate component of the midpoint in the X-axis direction of the image and the coordinate component of the midpoint in the Y-axis direction. Even in this case, the length of each coordinate component from the reference point to the predetermined point can be used as normalization information for specifying the position of the indication point when the image has a specified size.

  The coordinate information transmitted from the portable terminal 40 to the MFP 10 includes standardized information for specifying the position of the indication point when the image has a specified size regardless of whether the image is enlarged or reduced (see FIG. 7 (a) enlargement factor k and (Xe-Y, Ye-Y) in FIG. For example, instead of FIG. 7A, coordinate components (AX, BY) having the drawing start coordinates as the origin and the enlargement factor k as the normalized information may be transmitted to the MFP 10 separately. Good. Even in this case, the MFP 10 can specify the position of the indication point when the image has a specified size. Further, information transmitted from the portable terminal 40 to the MFP 10 is not limited to information including standardized information. That is, if enlargement / reduction processing is not performed on the portable terminal 40, the MFP 10 can specify the position of the indication point when the image has a specified size without including the processing.

  The means for requesting the LCD data from the MFP 10 is not limited to that using a lifting operation on the touch panel 46 as a trigger. For example, a pressing operation on the touch panel 46 may be used as a trigger.

  The method for requesting LCD data from the MFP 10 is not limited to the method using an input event to the touch panel 46 as a trigger. For example, the LCD data may be requested at a predetermined time period. At this time, the screen ID is further transmitted from the portable terminal 40 to the MFP 10, and the MFP 10 displays the screen ID of the image displayed on the display area 44 a of the LCD 44 of the portable terminal 40 and the image displayed on the display area 20 a of the LCD 20 of the MFP 10. The LCD data may be transferred to the portable terminal 40 on condition that the screen ID does not match.

  The MFP 10 is not limited to the one provided with the LCD 20 and the touch panel 22. Even if these are not provided, if image data representing an image for guiding an input operation on the touch panel 46 is transmitted to the mobile terminal 40, the MFP 10 is operated in accordance with the input operation on the touch panel 46 of the mobile terminal 40. be able to. In this case, the MFP 10 cannot set an image displayed in the display area 20a of the LCD 20, but if an image for guiding an operation that can be accepted in the MFP 10 is set, the screen ID is checked. It is also possible to execute the processing of S128 to S140 in FIG. In this case, as a situation where it is determined that the screen IDs do not match, for example, when the mobile terminal 40, the MFP 10, or communication between the mobile terminal 40 and the MFP 10 is mixed with noise, or the mobile terminal 40 is temporarily There may be a case where a malfunction occurs due to a power shortage.

  The input unit (terminal side input unit, device side input unit) is not limited to the touch panels 22 and 46. For example, a device that detects pressing, lifting, dragging, and the like on the LCDs 20 and 44 by image recognition may be used.

  A communication method between the portable terminal 40 and the MFP 10 when operating the MFP 10 with the portable terminal 40 is not limited to Wi-Fi communication. For example, communication using the telephone network communication unit 50 may be used.

  The operation target (processing device) using the portable terminal 40 is not limited to the MFP 10. For example, you may operate the energy management apparatus of the house with which the house was equipped. In this case, if the telephone network communication unit 50 is used, the user activates an air conditioner or the like prior to returning home from a land far away from the house, or uses the power generated by the solar panel in preference to boiling water. It is also possible to give instructions. At this time, when the energy management apparatus itself includes a touch panel, it is possible to issue an instruction from the destination by an operation according to the touch panel operation in the house.

  The communication terminal is not limited to a terminal having a function other than the function related to the operation of the processing apparatus such as the MFP 10 such as a function of a telephone call via a telephone network, and is a terminal specialized for the operation of the processing apparatus. May be.

  In the above embodiment, part or all of the processing (software processing) realized by the CPU 52 executing the operation support program may be changed to hardware processing by an ASIC or the like.

  The “process executed by the processing device” in claim 1 includes a process for executing each function such as a process for executing a copy function, a process for executing a facsimile function, a process for executing a scanner function, and the like. Replacement processing is also applicable. For example, when the Fax icon is selected on the screen of FIG. 2A, a process of switching to the screen (image) of FIG.

  The “specific information during setting of the processing apparatus” in claims 3 and 4 corresponds to the screen ID of the image data displayed on the MFP 10. As shown in FIG. 3, the screen ID is provided corresponding to each image data. Therefore, a screen ID is also provided for image data that is not displayed on the MFP 10.

10 ... MFP, 40 ... mobile terminal.

Claims (11)

A display unit having a display area; an input unit that detects a point on the display area that is a point designated by the input medium; a communication unit that communicates with the processing device; and a control unit. In an operation support program for operating the processing device based on an input to the input unit and executed by a communication terminal comprising:
For an image that guides an input operation using the input medium for instructing processing executed by the processing device, image data representing the image and specific information for specifying the image data are received from the processing device. Image receiving means;
Display control means for displaying an image represented by the received image data in the display area;
The coordinate information of the indicated point on the display area detected by the input unit in a state where the image is displayed by the display control means, the type of input event for the input unit, and the display area being displayed in the display area Input information transmitting means for transmitting, to the processing device, the specific information for specifying image data representing the image with respect to the image ;
Error receiving means for receiving from the processing apparatus information indicating that the specific information transmitted by the input information transmitting means is different from the specific information being set in the processing apparatus;
When the error receiving means receives information indicating that the specific information being set is different from the specific information being set, the image requesting the processing device for image data specified by the specific information being set in the processing device Request means;
Is executed by the control unit. Error receiving means for receiving from the processing apparatus information indicating that the specific information transmitted by the input information transmitting means is different from the specific information being set in the processing apparatus;
If information to the effect that differs from said particular information in said set by the error reception section is received, according to claim 1, wherein to execute the error display means for displaying to that effect on the display area, to the control unit Operation support program. Causing the control unit to execute image request means for requesting image data from the processing device;
The image receiving means is for receiving image data transmitted by the processing device in response to a request from the image requesting means,
The input information transmission unit transmits an input operation to the input unit during a period from when image data is requested by the image request unit to when the image data is received by the image reception unit. The operation support program according to claim 1 or 2, wherein the process is not executed. Causing the control unit to execute image request means for requesting image data from the processing device;
The image receiving means is for receiving image data transmitted by the processing device in response to a request from the image requesting means,
When the image data is received by the image receiving means, when an instruction to request the image data again is issued from the processing device, the control section is caused to execute the instruction receiving means for receiving the instruction,
The image request means, on condition that an instruction to request again by the instruction receiving means is received, any one of claims 1 to 3, wherein the requesting the image data to the processing unit again The operation support program described in 1. When an instruction to request again is received by the instruction receiving means, the control unit is caused to execute inquiry means for displaying a screen for inquiring whether to permit issuing the request instruction again in the display area,
The image request unit requests the image data again from the processing device when an instruction to be requested again is received by the instruction receiving unit and the inquiry result of the inquiry unit is permitted. The operation support program according to claim 4 . The display unit is a terminal side display unit,
The input unit is a terminal side input unit,
The processing device includes: a device-side display unit having a display area; and a device-side input unit that detects a point on the display area of the device-side display unit and designated by an input medium; With
Image data received by said image receiving means, operation support according to any one of claims 1 to 5, characterized in that it is intended to represent an image to be displayed on the display area of the device-side display unit program. A display unit having a display area; an input unit that detects a point on the display area that is a point designated by the input medium; a communication unit that communicates with the processing device; and a control unit. A communication terminal comprising:
Each means used as the object which the control part performs by the operation support program according to any one of claims 1 to 6 is provided as means realized by at least one of software processing and hardware processing. A characteristic communication terminal. The communication terminal comprising: a display unit having a display region; an input unit that detects a point on the display region that is a point designated by the input medium; a communication unit; and a control unit. In the processing device operated based on the input to the input unit,
Request receiving means for receiving a transmission request for image data from the communication terminal;
On the condition that the request receiving means receives a transmission request, for an image that guides an input operation by the input medium for instructing a process to be executed by the processing device, image data representing the image is transmitted in the communication Image transmission means for transmitting to the terminal;
Specific information providing means for transmitting specific information for specifying the image data to the communication terminal in accordance with the transmission of the image data by the image transmitting means;
Input information receiving means for receiving from the communication terminal the coordinate information of the indicated point on the display area detected by the input unit, and the type of input event for the input unit;
Specific information acquisition means for acquiring, from the communication terminal, the specific information for specifying image data representing the image with respect to the image being displayed in the display area in accordance with reception by the input information reception means;
Collation means for determining whether the specific information acquired by the specific information acquisition means matches the specific information set by the processing device;
An execution means for executing processing according to the coordinate information received by the input information receiving means and the input event, when it is determined by the matching means to match ;
If it is determined by the collation means that they do not match, an error transmission means for transmitting that fact to the communication terminal;
A processing apparatus comprising: A storage unit,
Provided that the image shown by the image data which has not been transmitted by the image transmitting means is represented in a state that is displayed in the display area before Symbol Table radical 113, a change in the image displayed in the display area of said surface radical 113 9. The processing apparatus according to claim 8 , further comprising storage control means for storing image data representing the image before the change in the storage unit. The image transmission unit transmits image data stored in the storage unit when a transmission request is received by the request reception unit when image data is stored in the storage unit. The processing apparatus according to claim 9 . The display unit is a terminal side display unit,
The input unit is a terminal side input unit,
A device-side display unit having a display area;
A device-side input unit for detecting a point on the display area of the device-side display unit and a point designated by the input medium;
Wherein the image transmission means, according to the image data representing the image displayed on the display area of the device-side display unit in any one of 0 claims 8 1 and transmits to the communication terminal Processing equipment.

Images