Add interactive Qt panel to VR scene #43

cpinter · 2018-10-03T16:22:50Z

A very flexible way for showing menus in VR would be to implement off-line rendering of any Qt widget in a texture that is shown on a VTK plane actor, which can respond to virtual mouse events generated by the laser pointer (currently called probe mode in VTK) in VR.

Tasks that need to be done, start with first and only move to next one once it is done.

Generate 2D texture with Qt of arbitrary QWidget. Done when you have a QImage or QPixmap containing a QWidget with some other widgets in it (for example a QFrame with a QVBoxLayout having a QCheckBox and a QSlider on it) so that the widget is never shown on the Windows desktop (i.e. everything happens in the background).
Show texture generated in the previous step on a VTK plane actor (for example vtkImageActor). Done when the Qt GUI is visible in the scene shown in VR (positioning can be arbitrary at this point).
Refactor VR interactor style according the new events design (Csaba)
Re-add laser pointer (called probe mode in vtkOpenVRMenu*), TBD
Send laser pointer events to Qt, re-render the widget if necessary, TBD
Create default "pipboy" style panel with basic settings. Show up attached to left forearm if left controller is turned outside 180(ish) degrees. Make panel look like it's obviously interactive (back side not visible, slight frame on the edges, etc.)

Useful UI panels (extend as needs arise):

"Main menu" widget that takes the user to the "apps" (below)
Subject hierarchy (show/hide)
Segment Editor (simplified)
Interaction settings: group objects (moving them under the same interaction transform), toggle selectable, etc.

Related issues

Related pull requests

ENH: Initial VR home widget implementation #100

pieper · 2018-10-03T16:30:48Z

💯 this will be a great feature!

This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43

jcfr · 2019-05-23T15:26:10Z

To follow up on this, @martinken recently added classes to VTK for adding Qt widget into the VR scene:

cpinter · 2019-05-23T17:12:57Z

@jcfr Perfect! @mohammadrashid0917 just started working on this. We'll re-adjust the plan to use these classes. Mohammad, please hang on. In the meantime you can look at these classes.

lassoan · 2019-05-23T17:27:51Z

@jcfr We were lucky this time that minimal/no work was duplicated, but we need to coordinate a bit more closely to make sure we avoid parallel efforts. How could we do that? Where can we find planned VTK virtual reality developments? Do you have a category in the VTK bugtracker, labs page, …?

jcfr · 2019-05-23T18:25:39Z

How could we do that?

The plan described at the top of this issue is outstanding. I should probably have done a better job communicating about it with folks working on OpenVR at Kitware.

Moving forward, I think sharing the plan like the one at the top of this issue on VTK discourse would help.

Internally, the folks working on OpenVR features know that I am coordinating effort on the Slicer side and I think it worked pretty well as I was able to follow up here.

That said, I agreed that the feedback loop could be shorter.

May be having a list of contact associated with key VTK area would help ?

Where can we find planned VTK virtual reality developments?

Considering that this type of feature are driven by customer needs and often planned and executed in a short time frame, i think it is challenging for us to communicate about this features with a high level of granularity and far ahead.

jcfr · 2019-05-23T18:49:37Z

To follow up, here is a Slicer PR backporting the specific VTK changes to Slicer/VTK fork. See Slicer/Slicer#1141

I have checked that it compiles on Linux, after someone check that Slicer/VTK compiles on Windows and macOS, it will be integrated.

These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43

Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43

Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43

Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43

This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43

These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43

Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43

Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43

Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43

Squashed commits: ENH: Create framework for registering VR modules Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43 ENH: Make VR home widget controls functional These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43 ENH: Add VR optimized Qt Style Sheet This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43 ENH: Add VR home widget WIP Its UI is more or less complete but events are not handled and style sheet is needed ENH: Add virtual widget from arbitrary Qt Widget !WIP!