JP4135487B2 - User interface device and portable information device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a user interface device and a mobile information device using such a user interface device. More specifically, the present invention relates to a data input method and an interface method suitable for a mobile device, that is, a device having neither a keyboard nor a mouse.
[0002]
[Prior art]
A major problem with new mobile and handheld devices is that it is difficult to interact efficiently with the device. The input capabilities of mobile devices are typically limited to pen input and touch screen, buttons and jog dial type controllers.
[0003]
The problem with these input methods is that touch screens that block the screen often require the use of a pen, but the use of the pen is often difficult due to the limited resolution of the touch sensor. That is. Also, interaction with the touch screen facilitates an interaction type based on direct sequential manipulation of GUI interface objects. For example, in order to enlarge the map, the user must sequentially repeat scrolling, positioning, and enlargement operations.
[0004]
Alternative devices have been proposed to help create a small but easy to use mobile device. For example, Japanese patent applications JP11-143606 and JP07-64754 disclose devices that accept a user's physical interaction with the device housing. Physical interaction includes changing the shape of the deformable part or tilting the device.
[0005]
Although such devices have been proposed, few attempts have been made to develop graphical user interfaces that utilize these devices. Little research has been done on how such interfaces are useful in basic interface tasks such as data scrolling, navigation, and browsing.
[0006]
Conventional data entry and user interface techniques currently used by most portable devices have been replicated from desktop graphical user interfaces or have been born from attempts to extend them. An example of a conventional data input and interface technique is disclosed in Japanese Patent Application JP2000-207088. These approaches are usually based on the use of a pen and mouse and are generally not suitable for small handheld devices such as mobile or portable devices.
[0007]
For example, traditional interactions with GUI interfaces such as those used in desktop computers and PDAs are based on the concept of a cursor or pointer. The cursor and pointer represent the current position on the display screen as a graphic. For example, to change the current position in order to select a different actionable icon, the user can use an input device such as a mouse or keyboard as shown in FIG. One of these must be specified directly. In this specification, such a task is called pointing or selection.
[0008]
In some cases, the desired element may not be visible on the screen, requiring the user to scroll through the content to find it. Such scrolling of content is a separate task from the above-described pointing (selection). Scrolling usually requires one of the following:
[0009]
a) special user interface elements such as scroll bars, or
b) Interface mode switching. For example, in the current GUI, content begins to scroll down when the cursor reaches the limit of the visible region of the content. Typically, in the conventional interface method, as shown by the numbers 1 and 2 in FIG. 1B, these operations are triggered by the movement of the pointer and the pressing of the mouse button.
[0010]
These pointing (selection) and scrolling methods are particularly inefficient and difficult to use in devices with a small display screen.
[0011]
[Problems to be solved by the invention]
There are several attempts to study new types of interfaces and data entry techniques suitable for mobile devices. However, most of these studies involve 3D data control (Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G., Sing, K., “Exploring interactive curve and surface manipulation using a bend and twist sensitive input strip. And interactive curve and surface manipulation using torsion-sensitive input strips) ", Proceedings of Symposium on Interactive 3D graphics, 1999, ACM. 111-118), or data scrolling (Rekimoto, J., “Tilting operations for small screen interfaces”, Proceedings of UIST '96. 1996, ACM. Pp. 167-168), or others (Fishkin, K. et al., “Embodied user interfaces for really direct manipulation ", Communications of the ACM, 2000, 43 ( ), 74-80 pages), and the like, have focused on a single task.
[0012]
Furthermore, text input using conventional mobile devices or handheld devices is quite problematic in the following respects. That is, the input functions of mobile devices are typically limited to pen input and touch screen, buttons and jog dial type controllers. There are currently three widely used methods for text input. That is, a system based on gestures such as a keyboard (on the screen or as a physical array of buttons), numeric pad input on a mobile phone, and Graffiti (trade name) from Palm Computing. Among these, the text input method using a numeric pad is probably the most widely used. However, such an approach has several drawbacks. This is because the button has to be pressed many times to enter each character.
[0013]
As the device size decreases, all of the conventional approaches described above become more difficult to use. The need for physical buttons limits the miniaturization of portable computing devices. The touch screen has a limited detection resolution, and there is a possibility that the user may block the screen during an input operation, so there are many problems in a small device.
[0014]
Furthermore, existing computer input devices such as a mouse and pen interface can perform various input operations such as pressing a button on a mouse or pressing a pen on a tablet. However, an external input device is required, which can be a problem for very small devices.
[0015]
The present invention has been made in view of the above-described conventional user interface and data input technique. Providing a data input and interface means for a device that interacts with the device by a user physically operating the housing of the device, and / or a visual display and a device including such data input and interface means desirable.
[0016]
In addition, graphic user interface means that allow a user to perform a wide range of tasks without using buttons, pens, or any other mechanical controller, and / or visual display and such graphic user It would be desirable to provide an apparatus that includes interface means.
[0017]
In addition, it is desirable to provide a data input and interaction means for a device that includes a two-dimensional position sensor and a one-dimensional analog sensor to detect user actions that command the device to perform tasks. In this specification, a task refers to a specific operation mode such as scrolling of an image displayed on a screen and selection of a part of the displayed image.
[0018]
In addition, it is desirable to provide a graphical user interface means for a device that includes a two-dimensional position sensor and a one-dimensional analog sensor for inputting text or selecting from a menu / list.
[0019]
[Means for solving problems]
An information processing apparatus according to the present invention includes a display for displaying information, and pressures from a plurality of directions applied to the information processing apparatus, and based on the pressures, the surface of the display applied to the information processing apparatus An analog sensor for detecting the direction and amount of pressurization in the normal direction, and based on the direction and amount of pressurization detected by the analog sensor , Information on the display As at least one of the transparency and size of the second view displayed above the first view as In response to the amount of pressurization, and the amount of pressurization detected by the analog sensor is equal to or greater than a predetermined threshold value. The display of the display is changed so that the display of the first view and the second view are relatively changed. At the same time, it has a control unit that controls not to make a change corresponding to the amount of pressurization until the amount of pressurization becomes zero.
The information processing apparatus according to the present invention may further include a two-dimensional position sensor for detecting at least one of a position touched by the user with respect to the information processing apparatus and a moving direction of the position touched by the user. it can. The predetermined display may be determined based on the position or direction detected by the two-dimensional position sensor.
The display may be constituted by a flexible display panel, and the two-dimensional position sensor may be disposed on the back surface of the display panel.
A vibration generating unit that generates vibration when the amount of pressurization exceeds a predetermined threshold can be further provided.
The information processing method according to the present invention includes a display for displaying information, and pressures from a plurality of directions applied to the information processing apparatus, and based on the pressures, the surface of the display applied to the information processing apparatus In an information processing method of an information processing apparatus having an analog sensor for detecting the direction and amount of pressurization in the normal direction, based on the direction and amount of pressure detected by the analog sensor , Information on the display As at least one of the transparency and size of the second view displayed above the first view as In response to the amount of pressurization, and the amount of pressurization detected by the analog sensor is equal to or greater than a predetermined threshold value. The display of the display is changed so that the display of the first view and the second view are relatively changed. At the same time, it includes a step of controlling not to make a change corresponding to the amount of pressurization until the amount of pressurization becomes zero.
In the information processing apparatus and method of the present invention, based on the direction and amount of pressurization detected by the analog sensor , Display information As at least one of the transparency and size of the second view displayed above the first view as In response to the amount of pressurization, and the amount of pressurization detected by the analog sensor exceeds a predetermined threshold value. The display of the display is changed so that the display of the first view and the second view are relatively changed. At the same time, control is performed so that no change corresponding to the amount of pressurization is performed until the amount of pressurization becomes zero.
In accordance with one embodiment of the present invention, a user interface device having a flexible portion is provided. The user interface device includes an analog sensor for detecting deformation of the flexible portion, and one of a plurality of first type input states based on the deformation detected by the analog sensor. And a means for executing the task, wherein the task is related to the selected first type input state, and each of the plurality of first type input states is the possible state. Associated with different deformation states of the flexible part.
In this embodiment, at least one of the plurality of first type input states is associated with one of dynamic and static positive deformations of the flexible portion, At least one of the first type input states is associated with one of the dynamic and static negative deformations of the flexible portion.
[0020]
In this embodiment, one of the plurality of first type input states can be associated with a neutral state of the flexible portion in which no deformation is detected. Preferably, at least one of the tasks is for controlling a graphical user interface object.
[0021]
The user interface device according to the present embodiment further includes a two-dimensional position sensor for detecting at least one of a position touched by a user in a two-dimensional plane and a direction of movement of the position touched by the user. Means for detecting a second type of input state related to a position touched by a user detected by the dimension position sensor and executing the task, the task being related to the selected second type of input state is doing. Further, at least one of the first type input states is preferably a transition state corresponding to a task that performs analog control of a graphical user interface object.
[0022]
The user interface device is configured as a single housing electronic device including a flexible display panel as the flexible portion, and the two-dimensional position sensor is located on the back side of the flexible display panel. It is preferable that they are arranged. Preferably, at least one of the tasks related to the second type of input state is for controlling at least one of the movement direction and position of the graphical user interface object. Alternatively, the aforementioned tasks associated with the second type of input state may be any of the shapes of a graphical user interface object or any other graphical object associated therewith, such as scaling, rotation, projection, etc. It is also possible to control the geometric transformation.
[0023]
According to another embodiment of the present invention, there is provided an apparatus configured to have a single housing including a processing unit and a display unit. The device includes an analog sensor disposed on the housing for detecting a user's analog input provided to the housing of the device. The processing unit changes a screen view displayed on the display unit based on an output value of the analog sensor.
[0024]
In the present embodiment, the screen view to be changed includes an image superimposed on an existing view, and the processing unit includes a visual characteristic of the image to be superimposed according to an output value of the analog sensor. One may be changed. Alternatively, the screen view to be changed includes a selectable item and an image that can give a visual impression to the user that the selected item is shown, and the processing unit outputs the analog sensor The selectable item and the selected item included in the image may be changed according to the value.
[0025]
Furthermore, in the present embodiment, the apparatus may include scroll means for controlling the scrolling of the screen view in accordance with a user input. The processing unit detects whether the position of at least one of the selectable graphic user interface elements displayed in the current screen view has reached a predetermined position in the screen of the display unit; In order to select the graphic user interface element and confirm the selection of the detected element, the operating mode may be switched to accept user input.
[0026]
In accordance with yet another embodiment of the present invention, a graphical user interface and data entry techniques are provided. The interface and data input method of this embodiment is suitable for devices having a display, such as mobile or handheld computing devices, PDAs, mobile phones, etc., based on using two-dimensional position inputs with one-dimensional analog inputs. Yes.
[0027]
These data inputs are obtained by using two types of sensors: a two-dimensional (2D) position sensor and a one-dimensional (1D) analog sensor. This interface technique allows the user to command common common tasks such as data navigation and menu selection. The two-dimensional position sensor may be disposed on the back side of the display screen or any other position that does not block the display screen. The two-dimensional position sensor may include a touch panel or any other device that can detect a two-dimensional input action provided by a user.
[0028]
The 1D analog sensor and the 2D position sensor may be integrated into the device itself. In this embodiment, the user does not use separate input devices and controls, but rather interacts with the device by operating the device itself, and the user observes the visual output on the screen of the display. . The following are typical examples of the apparatus according to the present embodiment.
a) A device comprising a flexible visual display that is embedded within the device and presents information to the user using single or multiple bend sensors that measure the bend at one or multiple points.
b) includes a non-flexible visual display attached to the device itself for presenting information to the user using pressure or force sensors that measure the user's force from multiple directions applied to the device apparatus.
c) In apparatus a) or b), an apparatus using an additional flexible (or non-flexible) single or multiple point sensitive two-dimensional position sensor attached to the apparatus.
[0029]
One of the important features of this embodiment is an interface based on physical operations on the apparatus main body such as bending, contact, sliding, positioning, etc., not based on a mouse and buttons. The present embodiment provides a simple interaction approach, which allows a simple but stable interface design for such an interaction paradigm.
[0030]
According to another embodiment of the present invention, a data selection technique for the interface of the previous embodiment is provided. The data selection method of this embodiment integrates the functions of data selection and data scrolling / navigation. This approach allows the user to display the entire content on the visual display, ie images and / or text, on the screen by scrolling in one or two dimensions using a 2D position sensor attached to the device. Interface elements can be selected. The operable interface element is selected by a specific algorithm (matching algorithm). Appropriate visual, auditory, and haptic feedback can also be provided when selection is performed.
[0031]
According to still another embodiment of the present invention, a selection confirmation technique for an interface according to the above-described embodiment is provided. In this embodiment, when selection and interaction are performed using one of the following methods, the selection may be confirmed using a 1D analog sensor.
[0032]
a) By detecting whether the output value of the 1D analog sensor has reached a threshold, this approach can be used, for example, by pressing a graphic button, or selecting a command from a list, or of a link in a web browser. The user can be made to select any interface data item, such as a selection.
[0033]
b) By using the two directions of the 1D analog sensor separately, it is detected whether the output value from the 1D analog sensor has reached a positive or negative threshold value. For example, it can be used to navigate back and forth in a hierarchical structure such as a file structure, menu structure, or hyperlink content. This can be done by defining negative and positive input directions on the 1D analog sensor. For example, it is possible to move down in a hierarchy using bending in one direction and move up in a hierarchy using bending in the opposite direction.
[0034]
c) Use the technique according to methods a) and / or b) but generate a smooth analog transition between views of the interface before confirming the selection (eg show next page or show dialog window). The transition can be controlled by the output value from the 1D analog sensor. Preferably, both the current and next interface views can be viewed simultaneously during a visual transition.
[0035]
Alternatively, by using the interface and data input method of the above-described embodiment and using the 1D analog sensor together with the selection confirmation method of the above-described embodiment, the additional information attached to the user interface element can be previewed. May be. The attached interface element may be help data, for example. Additional information can be displayed using the transparency and / or zoom techniques described above in conjunction with a 1D input sensor.
[0036]
Furthermore, the user interface and data entry techniques according to the above-described embodiments can also be used for simultaneous zooming and viewpoint control on visual data, or 3D navigation in a 3D user interface. The device according to the above embodiments may be provided with a haptic display for haptic feedback.
[0037]
In the above-described embodiment, a 2D position input is supplied using a 2D position sensor. Alternatively, if 2D position input is used for scrolling the screen view, the 2D position sensor may be replaced with any other type of sensor capable of scroll control. For example, a tilt sensor including an accelerometer may be used to detect the tilt of the device, and the user may control the scroll speed and direction by tilting the device in a certain direction and amount.
[0038]
According to yet another embodiment of the present invention, a graphical user interface control technique is provided. The graphical user interface control technique of this embodiment combines a two-dimensional position input and an analog input to allow selection from a three-dimensional array displayed on a visual display. This graphical user interface control technique may be used for text entry and menu / list methods.
[0039]
The graphical user interface control method of this embodiment is suitable for a mobile computing device (for example, a PDA, a mobile phone, etc.). The graphical user interface can select a group of items, then select a single item from the currently selected group, and then confirm or cancel the current selection. This selection is made using one of several possible combinations of simultaneous or discrete analog inputs and 2D position inputs.
[0040]
According to another embodiment of the present invention, an apparatus is provided that is configured to have a single housing that includes a processing unit and a display unit. The device includes an analog sensor disposed on the housing for detecting a user's analog input provided to the housing of the device. The processing unit includes an image processing unit having a plurality of operation modes for generating a screen view displayed on the display unit. The processing unit controls at least one functionality of the operation modes based on the output value of the analog sensor. For example, the image processing unit may include a moving image playback operation mode, and may control the speed of the playback operation using an analog output.
[0041]
According to yet another embodiment of the present invention, an apparatus is provided that includes a user interface unit. The user interface unit includes a user interface device according to any one or combination of the above-described embodiments, or utilizes the user interface technique. The user interface unit further includes one or more additional input devices. The additional input device may be an input device using a keyboard / button / switch, a pointing device, a mouse, a touch screen, a voice controller, a remote controller, a joystick, or the like.
[0042]
DETAILED DESCRIPTION OF THE INVENTION
[First Embodiment]
(1) System configuration and components
FIG. 2 is a block diagram of the mobile device of the present embodiment. The mobile device of this embodiment includes a one-dimensional (1D) analog sensor 102, a two-dimensional (2D) position sensor 105, a display 106, an analog signal acquisition unit 107a, a position signal acquisition unit 107b, a processor unit 108a, and a data storage device. 108b and an image processing unit 108c.
[0043]
In this embodiment, the processor unit 108a may include a processor (CPU), a memory, and all necessary units for controlling the information processing, the display 106, and the sensors 102 and 105. The mobile device may further include any wired or wireless communication unit for communicating with other devices and / or network connections.
[0044]
The user uses the 1D analog sensor and the 2D position sensor to control the mobile device and input data. These sensors may be arranged at any position as long as they do not block or interfere with the screen of the display 106.
[0045]
A plurality of 1D analog sensors may be provided to detect physical operations of the mobile device in different directions. For example, a set of 1D analog sensors may be provided to detect user bending inputs about the x and y axes, respectively. It is advantageous to have such a 1D analog sensor set if the display of the device can be used in both portrait and landscape modes. Even if the user performs the same bending gesture for both modes, the apparatus can detect in which mode the current bending input is performed.
[0046]
A 1D analog sensor produces an analog change in a control parameter of a measurable output signal, such as voltage or resistance. That is, these parameter changes directly correspond to changes in force applied to the 1D analog sensor. A 1D analog sensor may include one or more sensing devices attached to the apparatus. Sensing devices include a force sensing device that detects applied force, a bending sensing device that detects a shape change, a pressure sensing device that detects applied pressure, a rotation sensing device that detects rotational motion, and a 1D analog sensor It may include any other sensing device that detects changes that have occurred in connection with physical operations performed on the.
[0047]
In the present invention, the detection device is not limited to any specific type, and can be an electrostatic type, an electromechanical type, a magnetic type, an optical type, and a physical type thereof as long as an appropriate parameter can be detected. Various types of sensing devices that use a combination of properties can be used.
[0048]
In response to a user operation for specifying the position of the GUI element displayed on the display 106, the 2D position sensor outputs a signal indicating the position or a change in the position. The 2D position sensor 105 may be a 2D tracking sensor, a 2D touch sensor, or any other sensor or group of sensors capable of detecting a two-dimensional change. The 2D position sensor may include a flexible mesh sensor or a capacitive type.
[0049]
The user input detected by the sensors 102 and 105 is acquired by the analog signal acquisition unit 107a and the position signal acquisition unit 107b, respectively. Data acquired by the acquisition units 107a and 107b is processed using the processing unit 108a, information stored in the data storage device 108b, and the image processing unit 108c, and image data for updating the graphical user interface elements. The corresponding graphic image data is generated. The user input of this embodiment can be interpreted as an event in a conventional object-oriented, event-driven programming language. That is, when an event is detected, an event handler (program) is activated in response to the detected event, and a desired GUI operation is executed.
[0050]
The information stored in the data storage device may include various images or map data elements corresponding to the images displayed on the display 106. The details of the processing steps performed in the various tasks are described below. Finally, the display 106 is used to present an image of the updated graphical user interface element to the user.
[0051]
Alternatively, in addition to the visual display, a haptic display and / or an audible display (audio output) may be presented to the user. A haptic display and an audible display may be provided for haptic and audible feedback of user input.
[0052]
3A to 3D show examples of the mobile device of this embodiment. FIGS. 3A and 3B represent the configuration of a mobile device that includes a flexible display 106 having an embedded bend sensor 102 and a flexible 2D position sensing panel 105 on the back side. 3 (A) and 3 (B) show two shapes of the mobile device in this example when a force is applied in the opposite direction.
[0053]
In this example, the flexible display 106, the bending sensor 102, and the flexible 2D position detection panel 105 function as the display 106, the 1D analog sensor 102, and the 2D position sensor 105, respectively. The flexible display may be a display composed of flexible organic electroluminescence or any flexible member.
[0054]
User interaction with the mobile device may include:
a) Bending the entire apparatus up and down and viewing the input result detected by the bending sensor 102 on the screen of the flexible display 106.
b) While bending the entire apparatus up and down, the position of the interface element is controlled using the 2D position detection panel 105 on the back side.
c) Bend the entire device up and down in a pattern that is recognized by the device and can be used to issue commands.
d) The apparatus is bent to some extent, and the apparatus is continuously bent by that amount, and the rate of change is specified for a parameter with the amount of bending.
[0055]
3 (C) and 3 (D) show one alternative implementation of the above configuration, in which it is impossible to bend the device itself, but the user can either A force is applied to the pressure pad 103 disposed on the surface. The user interaction is similar to that of the example shown in FIGS. 3 (A) and 3 (B), and in either case, the user can enter a similar method to enter data and control the device. Apply force to the device.
[0056]
(2) Definitions and terms used
(2.1) Interpretation of data from 1D analog sensors
In this document, a generic method for referencing data received from a 1D analog sensor is used. As described above, a 1D analog sensor can be a variety of sensors that return a numerical value proportional to the user input provided to the sensor. Although various sensors or combinations of sensors can be used, the following description often uses force sensors as 1D analog sensors.
[0057]
Further, herein, a 1D analog sensor may be referred to as an analog controller or analog input device.
[0058]
Suppose that the measured values of the analog control variables from the 1D analog sensor N are distributed within a certain range, and we set a certain threshold that can be, for example, -100 or +100. Next, as shown in FIG. 4, the data from the sensor is interpreted as follows.
0 (eg, null): “Neutral” state of the device (403), ie no input from the user.
N = 100: The user applied input, and the sensor value reached the threshold value or the target value. This event is called “upward target” (401).
N = -100: Same as above, except that the negative threshold is reached. This event is called “downward target” (405).
N varies from 0 to 100. The user moves the analog controller to change its output value between neutral and a positive threshold, for example in a positive transition space. This event is referred to as “upward transition” (402).
N varies from 0 to -100. Same as above, except that the user moves the analog controller in a negative transition space. This event is called “downward transition” (404).
[0059]
Note the following: a) Some sensors may only have a positive space area. For example, the pressure sensor can detect data only in one direction. b) A combination of sensors can be treated as a single sensor. For example, it is possible to detect the positive and negative directions of an input by combining two pressure sensors. And c) combining one sensor for detecting only the amount of force applied by the user and one switch for detecting the direction of the force.
[0060]
(2.2) Interface elements on the screen and their interaction
In this specification, embodiments of the present invention are described using selectable or operable on-screen elements. These are graphical user interface elements that can interact with the user. In most cases, the interaction with the selectable interface element consists of two stages.
[0061]
First stage: selection of interface elements that the user wishes to interact with. In many cases, there are many selectable interface elements on the visual screen at the same time, so the interface must provide the user with a tool that specifies which elements the user wants to interact with. If only one interface element is controlled at a time, the selection page can be omitted.
[0062]
Second stage: interaction. After the user selects an interface element, the user can interact with the selected interface element. The type of interaction can vary and depends on the type of interface element. In some cases, the interaction is simply a confirmation of a selection to perform an appropriate action, such as pressing a button, selecting a file name to load from a list, or a link to follow on a web page. Others may be more complex, such as moving graphic interface elements on the screen.
[0063]
(3) Examples
The following describes examples of user interfaces and / or data entry techniques that can be used in a mobile device according to an aspect of the present invention. In the following example, the mobile device is assumed to include an auditory display and a haptic display in addition to the configuration shown in FIG. For the sake of generality, in the following example, the 1D analog sensor is not limited to any particular type; instead, the user observes the output on the visual screen of the display 106 while 2D It is assumed that positioning and 1D analog input can be controlled. The type of 1D analog sensor is not envisioned, but the 1D sensor detects the amount and direction of the user's bending input made to the mobile device.
[0064]
(3.1) Integration selection and data browsing interaction method
According to this example, an interaction method is provided that integrates 2D or 1D browsing and selection of content such as text or images into a single task. In this interaction method, the user simply scrolls the content on the screen of the display 106 in one or two dimensions using the 2D position sensor 105 (eg, using the touch screen on the back of the device). The selectable interface elements are automatically selected depending on their current position on the screen, and the relationship between the position on the screen and which element is selected is defined by a matching algorithm. The matching algorithm will be described in detail below. Thus, to select a selectable interface element on the screen, the user simply scrolls the content so that the desired element is selected.
[0065]
FIG. 5 shows a flowchart and snapshot illustrating the example process for selecting interface elements by content browsing. The right side of FIG. 5 shows the basic steps of this process, and the left side of FIG. 5 shows a screen shot of the application and received data.
[0066]
Step 51: The user browses the subway map on the screen of the display 106. The currently selected item is item 201.
[0067]
Step 52: Using the 2D position sensor 105, the user scrolls the map to the left by X and Y units.
[0068]
Step 53: The actual left scroll amount depends on the application and the 2D position sensor 105 and is usually changed using several mapping functions (eg, sensitivity can be adjusted). Accordingly, the actual scroll amount is calculated by F (X) and G (Y), which are mappings between the data received from the input device (2D position sensor 105) and the actual scroll amount of the content. The map is then scrolled by this content.
[0069]
Step 54: The processor unit 108a then uses a matching algorithm to search for the most appropriate selectable interface element for the content location on the screen.
[0070]
Step 55: If a selectable interface element is found, the process proceeds to step 56. If not found, go to step 58.
[0071]
Step 56: Select the newly found selectable element shown as 202.
[0072]
Step 57: Remove visual feedback for the previous element 201 and use various visual and haptic feedback to indicate the new selected element 202 as the active element.
[0073]
Step 58: If the currently selected element is still visible, the process proceeds to step 51, otherwise the process proceeds to step 59.
[0074]
Step 59: Remove all indications for the currently selected element and reset the visual interface to the null configuration.
[0075]
The matching algorithm defines which selectable or actionable item should be selected from several visual items depending on the current content position on the screen. A wide variety of algorithms can be used. For example,
a) Select the item closest to the center of the screen. To select an item under this matching algorithm, the user scrolls the content so that the desired item is in the center of the screen.
[0076]
b) Select an item in a direction opposite to the user's scroll direction. To select an item under this matching algorithm, the user scrolls the content in the opposite direction to the scroll. For example, the items to be selected are attracted.
[0077]
c) The item to be selected is identified by a combination of the matching algorithms described in a) and b) above or by any other matching algorithm that allows identification of a single item to be selected according to user input. The
[0078]
(3.2) Analog selection input method
In the conventional method, a button pressing operation is performed on a physical input device such as a mouse or a keyboard to confirm selection of a user interface element. Such a button pressing operation is a discrete event. This example describes several techniques for confirming data selection and interacting with data on a mobile device. The mobile device may be a handheld device that includes an analog sensor attached to the handheld device and does not use any buttons.
[0079]
These techniques are primarily for the mobile devices described above, but these techniques can be applied to other devices with similar characteristics that can control both 2D position input and 1D force input simultaneously, for example. Is also applicable.
[0080]
(3.2.1) Simple confirmation
FIG. 6 depicts a simple selection case using the present technique for web page navigation, including a flowchart and a snapshot of the interface visibility. The bar 301 on the left side of the snapshot shows the output value of the 1D analog sensor, for example, the amount of user bending input.
[0081]
Step 61: The example technique starts when an operational item (eg, item 203) of an interface on the current interface view 204 of the screen is selected and the 1D analog sensor 102 is in its “neutral” state. .
[0082]
Step 62: Measure the current value of the analog controller, for example the amount of bending.
[0083]
Step 63: The processor unit 108a determines whether the analog controller has reached a target state (for example, an upper target or a lower target) as shown in FIG. If the target state is reached, the process proceeds to step 64. If not, the process returns to step 62 to repeat the measurement of the output from the 1D analog sensor 102.
[0084]
Step 64: Confirm selection and perform appropriate action such as loading new view 205. In this example, the next web page is loaded.
[0085]
Step 65: Update the visual interface for the next web page to provide haptic, visual and audio feedback to support this action.
[0086]
Step 66: Wait from the interface to return to the neutral state and continue from Step 61.
[0087]
(3.2.2.) Two-way confirmation of selection
Two-way confirmation works in the same way as the simple confirmation described in Section 3.2.1, except that it distinguishes between “upper target” and “lower target” states and uses them for different purposes. . For example, a “downward target” event can be used to confirm the selection, while an “upper target” can be used to return to the previous position.
[0088]
(3.2.3) Integrated preview and selective activation
The “upward transition” and “downward transition” states can be used to make a smooth transition between user interface views. In this interface, when the next interface element is selected, the transition takes place immediately, but the value of the analog controller from the 1D analog sensor 102 during the neutral and “upper target” state or the neutral and “lower target” state. Can be used to design smooth visual transitions between different user interface views. In this specification, this technique is called “analog link”.
[0089]
FIG. 7 shows a case of selection using this method for map navigation (similar to FIG. 5), and includes a flowchart and a visual state of the interface. FIG. 7 shows how two views can be overlaid. The initial view (current interface view 207) is a subway map, while the other view (next interface view 208) is a corresponding road map.
[0090]
Step 71: This approach begins when the 1D analog sensor 102 is in its “neutral” state and an operational item of the interface (eg, item 206) is selected on the current interface view 207.
[0091]
Step 72: Determine the current output value of the analog controller, for example, the bending amount.
[0092]
Step 77: If there is no input, the process returns to Step 72. If so, the process proceeds to step 78.
[0093]
Step 78: Overlay the next interface view 208 on top of the current interface view 207. The higher the value of the analog controller, the easier it is to see the next interface view of the user interface. The views can be overlaid using any one or combination of the following methods.
(A) Transparency: The next view is overlaid on the current view, and the transparency of the next view is controlled by the value of the analog controller. As the input applied by the user to the analog controller increases, the transparency of the next view decreases.
(B) Zoom: The size of the next view is controlled by the value of the analog controller. As more input is applied to the analog controller by the user, the next view becomes larger.
(C) A combination of zoom and transparency.
(D) Any other method that allows the user to view the two views at the same time, but can superimpose the two views in whole or in part.
[0094]
Step 73: Check whether the analog controller has reached a target state (eg, upper target or lower target). If the target state has been reached, the process proceeds to step 74. If the target state has not been reached, the process returns to step 72.
[0095]
Step 74: Confirm selection and take appropriate action. For example, the next user interface view 208 is fully loaded and the previous view is not visible.
[0096]
Step 75: Update the visual interface to provide haptic, visual, and audio feedback.
[0097]
Step 76: Wait for the analog controller to return to the neutral state and continue from step 71.
[0098]
(3.3) Method of previewing additional information by interaction
The “upward transition” and “downward transition” states can be used to provide the user with information related to the actionable item. Examples of information to be attached can be a preview of content in the link, attached help information, notes attached by the user, and the like. When the user applies input to the sensor, the “upward transition” or “downward transition” state provides the user with relevant information on the link. The characteristics of the additional information can be controlled by an analog link and may include:
a) Permeability.
b) Size.
c) Detailed information level. For example, as the input that the user applies to the analog controller increases, more information is presented.
d) A combination of the above or any other applicable properties.
[0099]
FIG. 8 represents a flowchart of a process according to the present example and a snapshot associated with the process when previewing the annotation data attached to an operable item using the present example technique.
[0100]
Step 81: If the 1D analog sensor is in its “neutral” state and the interface operational item (eg, item 209) is selected on the current interface view displayed on the screen, the process Start.
[0101]
Step 82: Measure the current output value of the analog controller, for example, the bending amount.
[0102]
Step 83: If there is no input, the process returns to Step 82. If there is an input, go to step 84.
[0103]
Step 84: If any viewable data is attached, the process proceeds to step 85. If there is no data, the process returns to step 82.
[0104]
Step 85: Overlay the attached data 210 on the current view. As the output value of the analog controller increases, the visibility of the attached data increases. The visibility control parameters may include the following.
a) Transparency: Data is overlaid on the current page, and its transparency is controlled by the output value of the analog controller. As the input applied by the user to the 1D analog sensor increases, the transparency of the additional data decreases.
b) Zoom: The size is that of additional information attached to the device controlled by the output value of the analog controller. As the input applied by the user to the 1D analog sensor increases, the size of the additional data increases.
(C) A combination of zoom and transparency.
(D) Any other method that allows the user to view two views simultaneously and allows the user to superimpose the two views.
[0105]
(3.4) Interaction method for zoom and viewpoint control
In this example, the GUI interaction method allows the visual information presented on the screen to be enlarged and reduced, and at the same time change the viewpoint position. The image is enlarged in the “upward transition” state and reduced in the “downward transition” state. The output value of the analog controller is mapped to the zoom speed or amount. Position control can be achieved through the use of a 2D position sensor, independent of analog input via a 1D analog sensor, so that the user can simultaneously zoom and view position control with the same single gesture. can do.
[0106]
(3.5) Three-dimensional navigation method
The method described in the above example (3.4) can be used for 3D navigation on the screen. Zooming in results in flying inward, while direction control allows control of the flight direction.
[0107]
FIG. 9 shows a flowchart of a process according to the present example and an associated snapshot of the process for an interface approach for 3D navigation using a mobile device according to the present embodiment.
[0108]
Step 91: The process begins when the 1D analog sensor 102 is in a neutral state and a 3D environment is presented on the screen of the display 106.
[0109]
Step 92: Measure the current output value of the 2D position sensor 105.
[0110]
Step 97: If there is no input, the process proceeds to Step 92. If there is an input, the process proceeds to step 93.
[0111]
Step 93: Update the 3D direction of the vector V that defines the direction of the motion 211 using the two-dimensional coordinate data output from the 2D position sensor. The vector may be presented to the user to provide visual feedback to the user or hidden.
[0112]
Step 94: Measure the current output value of the analog controller.
[0113]
Step 98: If there is no input, the process proceeds to Step 92. If there is an input, go to step 95.
[0114]
Step 95: Calculate the amount of translation D along vector V using the analog controller measurements. Various mappings may be defined, such as linear mapping, non-linear mapping, and any other.
[0115]
Step 96: Translate the viewpoint in the direction of vector V by the amount D and continue from step 92.
[0116]
Depending on the viewpoint position and the controller parameters, variations of this 3D navigation algorithm can be implemented in various types of applications.
[0117]
In the above example, the 2D position sensor 105 is used to provide 2D position input. Alternatively, if the screen view is scrolled in either 1D or 2D direction using 2D position input, the 2D position sensor 105 is replaced with any other type of device that can control scrolling according to user input. You may do it. For example, a tilt sensor that includes an accelerometer can be used to detect the tilt of the device and allow the user to control the scroll speed and direction by tilting the device in a certain direction and amount.
[0118]
(3.6) Force feedback
The above example may be extended to use a haptic feedback device to communicate the current state of the device to the user. For example, haptic feedback may be realized by using a multilayer actuator.
[0119]
According to this embodiment, a user can interact with a mobile device or a small handheld device without using a mouse, touch panel and / or keyboard. If the touch panel is placed behind the display screen of the device, the user's interaction with the device will not block the screen, thereby facilitating the interaction. Interaction becomes simpler, more effective, more intuitive and fun.
[0120]
Furthermore, according to this embodiment, the interaction method with the GUI element can provide a simple and basic design method of an interface that can be used with a flexible computer and many new interaction devices. This approach can accommodate most of the basic interaction tasks with handheld computers.
[0121]
The advantage of the approach according to this embodiment is that it is a single integrated method that performs many interaction tasks by simple physical manipulation of the device. This makes the interface easier to use and more intuitive. The user only has to remember how to use some simple actions, and this is all that is needed to do the task.
[0122]
In this embodiment, no additional graphical user interface elements are required on the screen to perform tasks or interact with application programs. This therefore saves space on the screen and facilitates efficient use of the display area. Such a feature is particularly important for handheld devices having a relatively small display area.
[0123]
The technique according to the present embodiment can automate many tasks, compared to conventional techniques where small handheld devices are difficult to point for accurate interaction with a mouse or pen or other pointing device, Use of the device in mobile situations is facilitated.
[0124]
The approach according to this embodiment is more natural and easy to use and does not require an additional input device such as a pen. Therefore, this method is naturally suitable for a device in which a touch panel is arranged on the back side of the display.
[0125]
[Second Embodiment]
(1) System configuration and components
The second embodiment of the present invention will be described in detail below. The second embodiment is a method for text input and / or multi-dimensional menu method in a general GUI situation for the type of device described in the first embodiment as shown in FIGS. About. In the following description of the present embodiment, only system configuration parts different from those of the first embodiment will be described.
[0126]
The same definitions and terms are used for the interpretation of the 1D analog input data as shown in FIG. 4 of the first embodiment. One important property required for the 1D analog sensor 102 is that it returns to its neutral state when no input is provided by the user.
[0127]
The present embodiment can be applied to text input in various devices such as sentence input devices disclosed in Japanese Patent Applications (Publication) H10-154033 and H10-154144. Specifically, the text input method according to the present embodiment may be useful for selecting one of the items from the candidate item group displayed on the display screen.
[0128]
In the present embodiment, data from the 2D position sensor 105 is interpreted as follows. FIGS. 10A to 10C show an example of how 2D position data can be used to measure direction rather than position (as with a cursor). FIG. 10A shows an example of an item grid for selection, FIG. 10B shows an example of a direction segment, and FIG. 10C shows an example of a relationship between the direction segment and an item. Indicates. In this example embodiment, this approach is taken whenever a selection is made from a grid. The advantage of this approach is that the action is repeatable.
[0129]
In the 3 × 3 grid of items shown in FIG. 10A, the center item (number 5 in the figure) is selected by default. If this is the desired item, the user need only confirm this default selection. If one of the items around the center is desired, a selection can be made by “direction movement” using the 2D position sensor 105 described above. The directional movement associated with the array of angular segments (45 degrees in the example given below) is then measured. For example, the movement to the lower left will select the lower left item (number 7 in this example) regardless of the position of the user's finger on the 2D position sensor. The movement to the upper right selects the number 3 in this example, the movement to the right is associated with the number 6, and so on.
[0130]
(2) Details of interface method
(2.1) Interaction flow
The GUI control method according to the present embodiment operates in combination with the above-described input devices (1D analog sensor and 2D position sensor). Hereinafter, three variations of the GUI control method will be described. As an example, text input is used. However, the present invention is not limited to these variations, and the present embodiment is equally applicable to other variations in which items are not text characters but are menu items.
[0131]
FIG. 11 shows a basic flow of interaction. All variants of the GUI control approach share the following interaction steps, where steps a) and b) can be performed simultaneously.
a) Selection of items in the multidimensional array (steps 112-114).
b) Selection of one item within the selected group (steps 115-116).
c) Confirm or cancel the current group / item selection (step 117 / step 118).
[0132]
The interaction steps shown in FIG. 11 can be compared to the use of a telephone keypad. On the keypad, the user first selects a button representing a group of characters (ie, “2” to access “A”, “B”, “C”, etc.) and then the intended item (ie, a character). ) Repeatedly until it reaches. The GUI control method is different from this text input in that a 1D analog input combined with 2D position detection is used instead of a button.
[0133]
The embodiment of this example is different in the method by which the user selects an item group. All examples utilize the combined use of 1D analog input and 2D position input.
[0134]
(2.2) Exemplary embodiment using circulating layer
12 and 13 (A)-(D) show a flowchart of a process according to this example and a snapshot displayed on the visual screen of display 106 in connection with this process. Each of the snapshots shows a screen view 1300 where a window 1301 showing the entered text and a stacked layer 1302 are presented.
[0135]
This example uses a visual multi-layer system, each showing a group of items. At the “top” of the stack is the currently selected layer 1302b. The user cycles through these layers by manipulating the 1D analog sensor 102 that functions as a 1D analog input device. FIG. 13A is a snapshot of the start state in step 1201.
[0136]
-Group selection (steps 1202-1209)
FIG. 13B is a snapshot during group selection. In step 1202, a 1D analog input is measured, and in steps 1203-1205, the measured input is examined to see if the analog input is in the “up” or “down” direction or “neutral” state. If the 1D analog input is in the “down” state, the process proceeds to step 1209 to update the current group selection. If it is in the “up” or “neutral” state, proceed to step 1207 or step 1208, respectively. The analog input “downward transition” data is associated with the rate at which the application cycles through these layers. As the input data approaches the “down target” state, the rate of layer circulation increases. When the input device enters the “neutral” state, the circulation stops.
[0137]
Item selection (step 1211)
FIG. 13C is a snapshot during item selection. In this example, as shown in FIG. 13B, the items are arranged in a 3 × 3 lattice. Each time a new group is selected, its center item is selected by default. Other items are selected by direction input using a 2D position sensor (step 1211) and the updated current item selection is presented (step 1212).
[0138]
Confirmation or cancellation of selection (step 1210)
FIG. 13D is a snapshot during confirmation of selection. Once the desired group and item have been selected (yes at step 1207), the user confirms the selection by "transition up" (step 1210). If no selection is made, the “Done” layer must be selected to exit the selection mode.
[0139]
(2.3) Exemplary Embodiment Using “Ratchet” Layer
14 and 15 (A)-(F) show a flowchart of a process according to this example and a snapshot displayed on the visual screen of display 106 in connection with this process. As in the previous examples of FIGS. 12 and 13 (A)-(D), this example also uses a visual multi-layer system, each representing a group of items. At the “top” of the stack is the currently selected layer. The user selects a layer by operating the 1D analog input device. FIG. 15A is a snapshot in the start state (step 1401).
[0140]
Group selection (steps 1402 to 1409)
FIG. 15B is a snapshot during group selection. The analog input “down transition” data is mapped to the depth of the stacked layers. For example, for a stack of 4 layers and input data ranging from 0 (“Neutral”) to −5 (“Down Target”), the “Down Transition” data can be mapped as follows:
0 to -0.5 No selection
-0.5 to -1.5 Group 1
-1.5 to -2.5 Group 2
-2.5 to -3.5 Group 3
-3.5 to -4.5 Group 4
-4.5 to -5 Canceling group selection (step 1413)
When the 1D analog input device returns to the “neutral” position, the “deepest” group is selected until the group selection is canceled by the “down target” (step 1413) or the current selection is confirmed (step 1411). It has been done. FIG. 15C is a snapshot at step 1408 in which the default selection is performed in the “neutral” state.
[0141]
Item selection (steps 1412 to 1414)
FIGS. 15D and 15E are snapshots during item selection. As shown, the items are arranged in a 3 × 3 grid. Each time a new group is selected, its center item is selected by default. Other items are selected using the directional input by the 2D position sensor 105 as described above. Group selection and item selection can be performed simultaneously.
[0142]
Confirmation or cancellation of selection (steps 1406, 1411, 1413)
FIG. 15F is a snapshot during confirmation of selection. Once the desired group and item are selected, the user confirms the selection with the “transition up” state. The group selection can be canceled by the “down target” state, and the selection mode can be exited by the “up target” state.
[0143]
(2.4) Exemplary implementation using nested grids
FIGS. 16 and 17 (A)-(F) show a flowchart of a process according to this example and a snapshot displayed on the visual screen of display 106 in connection with this process. In the previous example, 1D analog input data is used as a direct means of group selection. In this example, 1D analog input data is used as a mode switch between group selection and item selection, while directional input using a 2D position sensor is used for both group selection and item selection. Visually, this system is represented as a group of 3 × 3 grids, each grid containing up to 9 (3 × 3) items, as shown in FIG. 17 (B). FIG. 17A shows a neutral state, from which the process starts (step 1601).
[0144]
Group selection (steps 1602 to 1613)
17B and 17C are snapshots during group selection. The “down transition” state of the 1D analog input detected by the 1D analog input device 102 triggers this group selection mode, in which one of the groups can be selected from the nested grid 1701. The central group is selected by default. While the 1D analog input device 102 is in the “transition down” state (step 1605), a group is selected using the direction input detected by the 2D position sensor 105 (steps 1610, 1609, 1613). The group selection can be canceled by the “downward target” state (step 1604).
[0145]
Item selection (steps 1612 and 1615)
FIGS. 17D and 17E are snapshots during item selection. While the 1D analog input device is in the “neutral” position, an item is selected using the directional input (steps 1612, 1615). The center item is selected by default. Item selection can be performed using the same algorithm as in the previous example.
[0146]
Confirmation and cancellation of selection (steps 1606, 1611 and 1614)
FIG. 17F is a snapshot during confirmation and cancellation of selection. Once the group and item are selected, this selection is confirmed by the “upward transition” state of the 1D analog input (step 1611). Any selection is canceled by the “down target” state (step 1614), and the selection mode is exited by the “up target” state (step 1606).
[0147]
In the example of the present embodiment described above, a direction input is input using the 2D position sensor 105, and one of the items in the selected group or one of the groups from the group set is selected. However, the present invention is not limited to these examples, and any other device can detect user input to select one of a selected group of items displayed on the screen. If so, it is possible to use such a device instead of the 2D position sensor 105. For example, a tilt sensor and its visual feedback may be used to determine direction input, ie, user input related to the direction relative to the center position of the visual screen for item selection.
[0148]
The GUI method according to the present embodiment described above allows a user to interact with a small handheld device without using a mouse, touch panel and / or keyboard. Since the 2D position sensor is located on the back side of the device, user interaction with the device does not block the visual screen, which makes it easy to maintain maximum visibility of the screen throughout the operation of the device It becomes.
[0149]
The GUI method described above is part of a wider GUI environment based on a combination of 1D analog devices and 2D position sensing input devices. Using this GUI method in this wider environment, the following functionality can be provided.
[0150]
a) Text input
According to the present embodiment, text input is possible without using a button or a pen. According to this embodiment, GUI means are provided to the device, and text or characters are entered more easily in mobile situations using analog and directional inputs.
[0151]
b) Menu style
According to the present embodiment, a menu method can be provided in an environment that does not use a point-and-click interaction based on a cursor. When this example implements the GUI method according to the present embodiment using text input, it is easy to assume a menu item instead of a character option. These menus function in much the same way as desktop GUI menus. That is, they may include items related to system functions ("Save", "Close", etc.) as well as items related to content (Bookmarks, TV channels, etc.).
[0152]
The first and second embodiments described above can be used to interact with a small handheld device or apparatus without the use of a mouse, touch screen placed on a visual screed, or keyboard.
[0153]
Examples of such devices and apparatus may include the following.
・ Remote control with display screen
-Remote control for television receivers.
PDA and personal information browser / partition.
・ Mobile phone.
·E-book.
・ Handheld game device.
・ Personal navigation device using GPS.
・ Remote display unit of TV receiver.
[0154]
Furthermore, the present invention can be applied to an apparatus for video image navigation. For example, an analog sensor in the “neutral” state plays the video at normal speed. Incorporating bending up and down can control the speed of video playback. Specifically, video playback and / or rewind operations are slowed when the analog sensor is bent up, and video playback and / or rewind operations are fast when the analog sensor is bent down. The control can be configured to Alternatively, control by analog output can include audio playback and / or rewind.
[0155]
In the first claim or the first description of this Japanese application, various aspects of the invention are described for the user interface so as to comply with the requirements of unity of invention under Article 39 of the Japanese Patent Law. The application is described with a focus primarily on the use of the flexible part as an analog sensor. However, the inventor believes that the subject matter disclosed herein has other inventive features in whole or in part. Waiver of patent rights with respect to such inventive features, or waiving the right to reacquire such inventive features as subject matter of various amendments, divisions, or continuation claims, etc. It is not what he intended.
[0156]
【The invention's effect】
In accordance with one aspect of the present invention, a user interacts with a device by physically manipulating the device's enclosure, and does not require a wide range of tasks without using buttons, pens, or any other mechanical controller. Data input and interface means to the device are provided so that
[0157]
In accordance with another aspect of the present invention, data input to a device utilizing position input and analog input to perform various tasks and instruct the device to enter text or select from a menu / list. And an interaction means is provided.
[Brief description of the drawings]
FIG. 1A is a diagram showing a conventional GUI control method for selecting a GUI element on a screen.
FIG. 1B is a diagram showing a conventional GUI control method for browsing / scrolling information.
FIG. 2 is a block diagram of an apparatus according to an embodiment of the present invention.
FIG. 3A is a perspective view of the apparatus shown in FIG. 2 having a flexible display device, a bending sensor, and a backside flexible tracking surface.
FIG. 3B: another perspective view of the device shown in FIG. 2 having a flexible display.
FIG. 3C is a perspective view of the apparatus shown in FIG. 2 having an inflexible monitor and a pair of pressure sensors.
FIG. 3D: another perspective view of the apparatus shown in FIG. 2 having an inflexible monitor.
FIG. 4 is a diagram illustrating a state of a 1D analog sensor and corresponding terms for describing the state.
FIG. 5 includes a flowchart of an algorithm for selecting GUI elements by content browsing and a snapshot of the visual screen at the steps of the algorithm.
FIG. 6 includes a flowchart of an algorithm for selecting a link using a 1D analog sensor and a snapshot of a visual screen with algorithm steps for an exemplary web browser application.
FIG. 7 includes a flowchart of an algorithm for an analog link where smooth transitions between user interface views using an analog controller are made and a snapshot of the visual screen at the steps of the algorithm. is there.
FIG. 8 includes a flowchart of an algorithm for previewing data attached to an actionable item and a snapshot of a visual screen at the steps of the algorithm.
FIG. 9 includes a flowchart of an algorithm for performing 3D navigation operations and a visual screen snapshot of the steps of the algorithm.
FIG. 10 (A): An exemplary item grid for use with directional input detected by a 2D position sensor for selection.
FIG. 10B is a diagram illustrating an exemplary direction segment used to detect a direction input.
FIG. 10C is a diagram showing the relationship between the direction divisions shown in FIGS. 10A and 10B and the item lattice.
FIG. 11 is a flowchart of a basic interaction flow in the second embodiment of the present invention.
FIG. 12 is a flowchart of an exemplary implementation using a cycle group according to the second embodiment.
13A is a diagram showing a snapshot of a visual screen in the start state of the flow of the example of FIG.
FIG. 13B is a diagram showing another snapshot of the visual screen during group selection in the flow of the example of FIG.
FIG. 13C is a diagram showing another snapshot of the visual screen during item selection in the flowchart of the example of FIG.
FIG. 13D is a diagram showing another snapshot of the visual screen during item confirmation in the flow of the example of FIG.
FIG. 14 is a flowchart of an exemplary implementation using a ratchet layer according to the second embodiment.
FIG. 15A is a diagram showing a snapshot of the visual screen in the neutral state of the flow of the example of FIG.
FIG. 15B is a diagram showing another snapshot of the visual screen during group selection in the flow of the example of FIG.
FIG. 15C is a diagram showing another snapshot of the visual screen in the neutral state by default selection in the flow of the example of FIG.
FIG. 15D is a diagram showing another snapshot of the visual screen during item selection in the flow of the example of FIG.
FIG. 15E is another snapshot of the visual screen during item selection in the flowchart of the example of FIG.
FIG. 15F is another snapshot of the visual screen during item confirmation in the flowchart of the example of FIG.
FIG. 16 is a flowchart of an exemplary implementation using a nested grid according to the second embodiment.
FIG. 17A is a diagram showing a snapshot of the visual screen in the neutral state of the flow of the example of FIG.
FIG. 17B is a diagram showing another snapshot of the visual screen during group selection (default selection) in the flow of the example of FIG.
FIG. 17C is a diagram showing another snapshot of the visual screen during group selection in the flow of the example of FIG.
FIG. 17D is a diagram showing another snapshot of the visual screen during item selection (default selection) in the flow of the example of FIG.
FIG. 17E is another snapshot of the visual screen during item selection in the flow of the example of FIG.
FIG. 17F: Another snapshot of the visual screen during item confirmation in the flow of the example of FIG.
[Explanation of symbols]
102: 1D analog sensor, 105: 2D position sensor, 106: display, 107a: analog signal acquisition unit, 107b: position signal acquisition unit, 108a: processor unit, 108b: data storage device, 108c: image processing unit.

Claims (5)

A display for displaying information,
An analog that measures the pressure from a plurality of directions applied to the information processing device and detects the direction and amount of pressurization in the normal direction of the surface of the display applied to the information processing device based on the pressure A sensor,
Wherein based on the orientation and amount of detected pressure by the analog sensor, wherein at least one of transparency and size of the second view to be displayed on the first view as the information of the display pressure The display of the first view and the second view is relatively changed in accordance with the amount of pressurization detected by the analog sensor being equal to or greater than a predetermined threshold. An information processing apparatus comprising: a control unit that performs control so as not to change corresponding to the amount of pressurization until the amount of pressurization becomes zero at the same time as changing the display on the display . A two-dimensional position sensor for detecting at least one of a position touched by the user with respect to the information processing apparatus and a moving direction of the position touched by the user;
The information processing apparatus according to claim 1, wherein the predetermined display is determined based on the position or direction detected by the two-dimensional position sensor. The display is constituted by a flexible display panel,
The information processing apparatus according to claim 1, wherein the two-dimensional position sensor is disposed on a back surface of the display panel. The information processing apparatus according to claim 1, further comprising: a vibration generating unit that generates vibration when the amount of pressurization exceeds a predetermined threshold value. A display for displaying information and pressure from a plurality of directions applied to the information processing device are measured, and based on the pressure, pressure in a normal direction of the surface of the display applied to the information processing device is measured. In an information processing method of an information processing apparatus having an analog sensor for detecting a direction and an amount,
Wherein based on the orientation and amount of detected pressure by the analog sensor, wherein at least one of transparency and size of the second view to be displayed on the first view as the information of the display pressure The display of the first view and the second view is relatively changed in accordance with the amount of pressurization detected by the analog sensor being equal to or greater than a predetermined threshold. An information processing method including a step of controlling not to perform a change corresponding to the amount of pressurization until the amount of pressurization becomes zero at the same time as changing the display of the display .

Images