Add a GUI solution to LWJGL #101
Comments
|
Also I have a running version of Themable Widget Library, which works really awesome with LWJGL 3 and kappa's lwjglx library. |
|
Do agree that TWL is a great option for rendering GUI on top of 3d content. It only supports LWJGL2 atm, though looking at the code base it doesn't seem all that much work to port to LWJGL3. However for standalone ui's, SWT is probably the best and most mature option out there. It integrates nicely with the native platform and is small and lightweight. It also strips down pretty nicely (to a few KB's, the Avian example are good examples of how small, less than a meg for SWT support and the Avian JVM!). So looking into getting LWJGL3 nicely integrated with SWT should be explored. IMO a much better investment then spending time on AWT or JavaFX. Interestingly, l33tlabs (who host TWL), work on a really interesting project called Bling. They've somehow integrated SWT with TWL! I think they also use LWJGL2. They already have a full IDE running on the same, so pretty far along. There is a pretty long video here explaining how it works and theres a quick preview of what it looks like here. I'm not sure what l33tlabs plans are but if they released the underlying SWT/TWL library or if something similar could be written, it could probably be the ultimate and complete solution for LWJGL3 i.e. a small lightweight API that can render ui inside OpenGL applications, create native platform widgets (like dialog's, system tray integration, native controls, etc) or even enable full blown native applications that can render opengl content inside it (e.g. particle editors, game engine editors, etc). |
|
I wonder if it would be possible to somehow render HTML+CSS (and possible code the UI logic in javascript via nashorn maybe) for HUD? Another thing that maybe could help is the project dukescript. |
|
I think using Nashorn and JavaScript would not be the problem; the library implementation and its API would be, as it sounds rather like the tasks of browser engines like WebKit or Blink and the domain of the Web platform in general. However it would sure be great. :) |
|
Yeah, that's the thing. I didn't mean to create something like WebKit, but if we could create a binding for it and use it to somehow render the HUD, that is what, IMO, would be awesome. |
|
This "awesomium" seems to be what you were looking for:
It also allows to render into an OpenGL surface (one can define its own surface factory). |
|
In my humble opinion its more of an issue if a GUI lib doesn't supports core OpenGL than not supporting LWJGL 3 (TWL is really nice but it's renderer is GL 1.1 exclusively). NiftyGUI is there, has a core OpenGL backend, but doesn't supports LWJGL 3 (mainly Display/Keyboard/Mouse related changes and some function name changes). Also another concern of mine is the state in which the GUI lib leaves OpenGL. This should be very well defined, ie, what state OpenGL needs to be before the GUI lib can do whatever it needs to do, and what state is OpenGL left with once it finishes. This is vital for efficient state tracking. |
|
Unlike @dustContributor I'm not concerned with supporting "core OpenGL" standing as much as if it fits in the "lwjgl wheelhouse" of libraries and functionality, and if the general API fits thematically with other, existing lwjgl API calls. I don't have a specific recommendation on how to do that, so long as I would be able to pass in some UI component hierarchy, and just tell it to "show the gui". What would be wrong with basically "copying" the AWT and making it render for lwjgl? |
|
Supporting OpenGL 1.1 and not >= 3.2 core is an issue if you yourself want to use a "modern" OpenGL context and functions therein, which would both not be possible in a single context. Either TWL would not function correctly because it used deprecated OpenGL functions or you would not be able to use "new" OpenGL functions. This is an issue with Mac OS X, which either supports OpenGL <= 2.1 if you don't ask for a new context specifically, or you would only get a "core" context if specifically asked for, but then TWL would not work. One solution to this problem, and also to the problem of touched OpenGL state, could be multiple sharing contexts. One "deprecated" OpenGL context for TWL and another one for your own rendering. Sharing between an "old" deprecated context and a "new" core/forward compatible context does work at least under Windows with Nvidia. However, only "requiring" OpenGL 1.1 is the smartest thing one can do for such a widespread library like TWL, in my opinion. Whenever you require anything higher than OpenGL 2.0 it will not work on some person's machine. Compatibility is a very big thing here! |
|
That sounds both slow and bound to hit driver specific issues. No, I'd rather handle the state change myself. At the very least it should have a renderer interface that doesn't assumes fixed-function pipeline working underneath, TWL has that issue in various places last time I checked, which makes it a pain in the butt to code a core renderer for. I still propose NiftyGUI as a better alternative, right now it just needs a couple of classes using a LWJGL 3 backend. I opened an issue about it a while ago nifty-gui/nifty-gui#304 There is literally no other GUI library that has a core renderer that I know of (for Java that is), thus why I think its an important matter. OpenGL 3.0 has been out there for 7 years, with hardware supporting it going as back as 2006 (GeForce 8xxx, Radeon HD2xxx), that was almost ten years ago. |
Do we even need anything special for SWT integration? Looks like @httpdigest already has a sample here. |
|
The only downside of SWT is that it still (even in the very latest nightly build) cannot produce a GL >=3.2 core context. I had a look at the sources and found that the WGL solution at least is devised but commented out... But this problem can be solved by letting LWJGL3 do the GL context creation. So we would need a cross-platform way to create GL contexts in LWJGL3 based on a window handle. This handle is easily API-accessible in SWT. |
|
I recall digging through the sources and finding no clear way to hijack the SWT's component so I could use it with my core context (although my experience with window management APIs is limited). Both JavaFX and SWT (since 4.2 I believe) are theme-able with CSS, which is a pretty cool thing to have. I wonder if we could nag SWT developers about this... EDIT: Still no interest in NiftyGUI? |
|
I am going to build an own GLCanvas, something like a |
|
So any update on lwjgl native gui rendering methods? |
|
@httpdigest is making good progress with SWT integration, see his latest update. I experimented a bit more with JavaFX and have a working OpenGL backend that runs on top of LWJGL. This may seem like a huge step forward, but it's not enough. It would be easy to implement context sharing with the JavaFX context and access the textures directly. The problem is, JavaFX is architected in a such way that the rendering layer has no idea which textures correspond to which GUI elements. I don't have a solution to this yet, I'll explore some more ideas when I have time. I don't know if nanovg bindings will make it to the final LWJGL 3.0 release, but I'll try. |
|
The latest nightly build has jawt bindings, which in theory allows for the AWT integration in LWJGL 2 to be implemented in terms of LWJGL 3. Sample code can be found here (simplest possible implementation, Windows-only). |
|
I've been thinking. Now that NanoVG bindings are up, it might be possible to use it as backend for other GUI libs that lack an OpenGL core context, like TWL for example. I'm looking at TWL's sources, seems a lot of work though -_- |
|
I completely forgot about libRocket: http:https://librocket.com/ Its C++ though. |
|
This is kind of late but a while ago I did manage to make TWL run with LWJGL 3 though it wasn't good by any means and TWL does use old OpenGL functions. I should also bring up that one of the main problems I was having was figuring out a good way to get the event data from GLFW and processes it. As there can only be one listener it would be easy to make a version that just splits that up however it gets more complicated for the end user and taking over isn't a very good option as its not very easy to detect if your listener has been removed and could lead to problems with the end users code. |
|
Sweet, it even leaves rendering up to the user by just giving you commands to execute, perfect. |
|
Hey, if at all interested, I'm making a library for lwjgl, but for a broader spectrum*, from scratch. It'll allow direct support for GLFW in LWJGL3, support for the latest OpenGL (4.5+), and a way to build shaders on the fly during runtime (in GLSL). The window component system will be made to support XHTML and XML based layouts for quick building of GUIs and easy management, so you could simply build it in a browser or other development environment. ( *By which I mean it's for more than just lwjgl. It'll support GLFW, but it'll support other GUI systems as well: AWT, SWT, Swing, etc. |
|
Hey! It would be great if you add bindings to this library NanoGUI |
|
NanoGUI is C++ and probably hard to wrap. But it could make sense to port NanoGUI to Java on top of lwjgl's NanoVG. |
|
SGL now has code in it |
|
https://github.com/glegris/nuklear4j Dafuq, never saw what O_o EDIT: Oh, SWIG : / |
|
LWJGL will very soon have Nuklear bindings, see the nuklear branch. There's already support for everything, but I'm working on a Nuklear+GLFW+stb_truetype demo to understand and properly annotate the API (nullability, autosize parameters, etc). |
|
SWT uses native OS widgets, why would one want to use that? Did anyone evaluate whether it is possible to extend Graphics2D class/override repaint() methods and thereby redirect Java Swing UI painting to OpenGL/NanoVG? Besides that, one could set JFrames as transparent and place them on top of the OpenGL window... :) However, the best solution would be some generic Java UI lib that can be plugged into NanoVG, ie. a rewrite of Nuklear. CSSBOX provides html/css rendering, though I'm not sure how efficient animations would be. And interactions? In general, I'm not sure how efficient GUI animations would be with pre-written GUI libs. It would be an interesting task to think about the required structure of such a library to efficiently support all sorts of GUI animations. BTW: because Vulkan support is now quite widespread, is NanoVG compatible with it? |
|
Hi guys, I'd like to propose also another possibility.. I'm doing a full jvm port of imgui here (nuklear is based on its concept), it's almost complete, text input/handling needs still to be addressed but all the rest is done plus, it's also libgdx compatible :) See the Screenshots Thread for some user creations. Also see the Mega screenshots for an idea of the available features.
|
|
I was reading some Java articles and it popped up that Gluon published release/support plans for OpenJFX. For those who don't know, JavaFX is being removed from core JDK, and it will be released separately. So to use it you'd need it to import it like any other dependency (maven, raw jars, etc). And now it will be called OpenJFX. https://gluonhq.com/javafx-11-release-and-support-plans/ They have a GitHub mirror of the OpenJFX repository and they say they pay attention to the issues there. https://github.com/javafxports/openjdk-jfx Moreover, they say they have their bug tracking on the JDK thingy (although the site works like ass for me and I cant find it) https://bugs.openjdk.java.net/secure/Dashboard.jspa Anyway, I'm wondering if that given the changes JFX is going through, if it would be possible to push for direct access to the GL internals from it? As a means to integration with other GL rendering. All the stack is open source now so you got in that repo their "Prism" rendering engine and all, plus the HLSL/GLSL shader transpiler (Decora I think?) and some other nifty things. In any case, they went a different route for it so instead of making bindings and "lower" the layer of native interop, they instead "raised" it by making the rendering part in C/C++ (doing all their multi platform context management on their own and all) and exposing higher level operations to Java. That doesn't sounds like it'd interact well with how LWJGL works but I wouldn't know. It gets complicated because OpenJFX uses three backends, OpenGL ES 2, Direct3D and software. So what happens if you want to use OpenGL Core? Or Vulkan? It's pretty complex, and I personally believe that The One True Way™ to do an UI lib for games is doing it like Nuklear does, handing out buffers of vertices/uvs/indices at rendering stage and have the user be responsible of displaying that to the screen, but I thought someone might be interested. |
|
@dustContributor Yes, OpenJFX being decoupled from the JDK opens up opportunities, you can read some details in this JGO post. (No work has been done yet, I grossly overestimated the free time I'd have this summer) |
|
I see you know already how to tackle it. I inspected a bit OpenJFX code and it seems to have a rather direct rendering, in the sense that it has fairly hard codes operations in a bunch of classes that do specific things. I'm guessing that achieving Nuklear-style of integration (ie, making it spit out vertices and texture bindings, API agnostic) would be pretty hard. |
|
Hello everyone again.
If anyone want to contribute, request any changes - you are welcome. |
I'll repeat what I said above (except longer!). I dont think having your own rendering backends is a good idea. You don't know how I, library user, do my rendering, you don't know what API I run, you don't know what version I run, and you really shouldn't. Things that did their own drawing like Scaleform on big AAA games were phased out, because you have a complete subsystem trying its best to group rendering commands, group state and shader changes, and drawing all of that efficiently, and then you stuff your code with some other renderer that has some other API with some other concepts that just doesn't fits on how your engine does its drawing. You have libraries like NanoVG, Nuklear and (now defunct) libRocket that are really really easy to integrate because they have a clear distinction between what is UI handling, what is rendering and what is input handling. Without the library setup code, this is my rendering loop for Nuklear: try ( MemoryStack mem = MemoryStack.stackPush() ) {
// setup buffers to load vertices and elements
final NkBuffer vbuf = NkBuffer.mallocStack( mem );
final NkBuffer ebuf = NkBuffer.mallocStack( mem );
nk_buffer_init_fixed( vbuf, tmpVertices/* , max_vertex_buffer */ );
nk_buffer_init_fixed( ebuf, tmpIndices/* , max_element_buffer */ );
// converts buffers to the layout specified in the config
nk_convert( context, commands, vbuf, ebuf, bufferLayoutConfig );
}
// Upload data to the GPU
glAttribBuffer.bind().updateUnsafe( tmpVertices );
glIndicesBuffer.bind().updateUnsafe( tmpIndices );
// Process task commands.
int offset = 0;
// Compose render tasks out of nuklear's draw commands.
for ( NkDrawCommand cmd = nk__draw_begin( context, commands ); cmd != null; cmd = nk__draw_next( cmd, commands,
context ) ) {
final int elmCount = cmd.elem_count();
if ( elmCount < 1 ) {
continue;
}
// Submit task to the queue.
// Matrices computed in shader.
renderer.addToQueue( taskBuilder.reset()
.type( RenderTask.Type.NUKLEAR )
.meshId( glVao.id )
.material( materialsById.get( cmd.texture().id() ) )
.vertexCount( elmCount )
.indicesOffset( offset )
.build() );
offset += elmCount * 2;
}I just ask for Nuklear to fill vertex/index buffers, upload them to the GPU, then compose render tasks that use them and that's it. I know that will best fit my rendering scheme because it is my rendering scheme. It will use the API I use, it will use the version I use, it will track state changes like I do, it will batch UBO updates like I do, and run without switching shaders unnecessarily like I do, and if I wanted I could make it use my instanced rendering by grouping by material first. I dropped things like NiftyGUI because literally the first time I use them it turns out that there was some GL context interaction that they didn't account for so I had to spend time debugging what state my renderer was leaving GL with, what state the UI lib really needed (that may or may not be what the documentation says) and what state the GL context ended being in that made things not work correctly. The GPU is a big fucking thing, wouldn't it be nice if you (lib author) didn't have to account for these things? Wouldn't it be a concern better suited for myself the library user since I know what API I target and how I want to use it? I'd say: Write an UI lib, or a renderer, but not both in the same thing. They're separate concerns, and big enough to warrant their own separate libraries. The same argument can be made for input handling but that'd make this too long. I have an input system that translates GLFW's stuff for my own scheme, and those events get pushed to Nuklear, Nuklear doesn't cares how do I map my keys, devices or anything, it just cares for being fed input events in a format it understands. At least that's my POV after trying out integrating JavaFX, Swing, TWL and NiftyGUI multiple times over time. Nuklear, being the pain in the ass of a library it is to use, was way, way easier to integrate. And since the API is so clean cut on that regard, it was easier to debug. I know when it renders because I render it. I know when it processes input because I tell it to. That said, I know UI is hard, that's why I'm always in the lookout for a lib that does it for me. I understand making such a simple integration point isn't easy, and may require rethinking a lot of abstractions. For example, it's really hard to move TWL as-is to such scheme, or even to using a core GL context (nevermind Vulkan) with it because not only believes it should draw itself, it believes it should draw itself like they did with OpenGL 1.1 in 1998, which for any kind of recent project is a problem. The right abstractions just aren't there. Moreover, I will argue that moving to such scheme not only opens up the integration possibilities, it has no drawbacks. Nothing prevents you from implementing a GL 1.1 renderer for Nuklear, or a Vulkan one, or a software renderer, nothing prevents you from supplying separately your own backends to draw it if you really really want to code an UI renderer. Hell, it wouldn't surprise me if data-driving it that way made the whole thing faster via handling flat buffers of draw commands and vertices. It just makes the whole ordeal way easier for everyone involved. I'm not writing this for you specifically anyway, so dont take it as a big ass critique of your code, I've only skimmed through your library and it probably does a lot of things right that aren't even on my mind right now. I'm just writing it for anyone who is interested in doing/using an UI library in Java for videogame (or any real-time graphics) purposes. |
|
I understand what you mean, @dustContributor, and of course big projects often use their own decisions that fit better to the project and work faster. I'm not saying that my decision is good, but I tried to make it as much extensible as it could be. Any way every solution is good for it's purposes. If you need best performance - you implement something in your own way, if you need fast development without pain - you can choose existing solution. 👌 |
|
Interesting new feature for JavaFX https://twitter.com/johanvos/status/1128627206295629824 |
|
Sounds like it still uses texture copies though. =( See: https://twitter.com/tomsontom/status/1128698978520977411 DriftFX still looks much better. Just wish it were more portable. |
|
Looks like some momentum is starting to build for integrating DriftFX into LWJGL. |
|
would there be a way to surround a 3d scene with a gui i made using a flat quad? |
|
Now that eclipse-efx/efxclipse-drift#9 has been closed, what's next for the integration branch (https://github.com/LWJGL/lwjgl3/tree/driftfx)? Is it available for testing as a snapshot build? |
|
The latest snapshot (LWJGL 3.2.4 build 4) includes DriftFX. It's a build of the wip_performance branch. Since the implementation is unstable, performance is suboptimal and only Java 8 is supported, |
|
Is there a way I can skip building this or get it to build with java11-openjfx installed? It can't seem to locate classes related to openjfx. |
|
Hey @gudenau, As of 92c88d4 there is support for JavaFX 11. The build should download the necessary OpenJFX jar files automatically via Maven. If it doesn't work for you, please open a new issue and paste the error you're getting. You can also skip the DriftFX bindings by setting |
|
Holy cow, that's awesome! |
|
Update on the DriftFX integration: I'm about to remove the DriftFX bindings from LWJGL, because vanilla LWJGL + latest official DriftFX work great together! Details here: eclipse-efx/efxclipse-drift#17 (comment) |
|
How do I run javafx and lwjgl on mac Now I just need to set -XstartOnFirstThread to run |
If you want create native sub or child windows ( dialogs ) if I understand correctly. You need understand native bindings like glfw3 with X11, WinApi or Cocoa. Example Example long subwindow = Subwindow.Create() If you want drop & drag native window to window - you need write engine with native Api. I recommend you JNA to help with glfw3 windowing manipulation. I hope you understand me. |
|
Thank you for your comment. I am now using DriftFX. That is way easier and I can run mulitple GL renderers in one or multiple windows. I have implemented DriftFX into threekt with multiple windows here: https://github.com/markaren/three.kt/blob/master/examples/src/main/kotlin/info/laht/threekt/examples/javafx/HelloDriftFX.kt |
|
I would like to point out for anyone reading this thread, https://github.com/SpaiR/imgui-java is a java binding to Dear ImGui, which is a great gui library. I use imgui-java in a lot of my projects. |
|
Can you take a look at RmlUi https://github.com/mikke89/RmlUi |
|
@enesaltinkaya I don't see a C api for it, only a c++ api. It will require a C api for mapping to LWJGL. |
|
If I were to wrap a C api around this library (somehow), what would be the next step for integrating into LWJGL? |
Nope. LWJGL does has docs on how to contribute and so on. https://github.com/LWJGL/lwjgl3/tree/master/doc After reading that, best thing you can do probably is pick some simple lib already in LWJGL3 and look how its bindings are done. I also made a bunch of questions about this in this issue here #428 that may help you, it was a long time ago though. |
|
Update: https://github.com/SpaiR/imgui-java seems to be the most stable at this moment The only nice thing that DriftFX seems to have and imgui not is having multiple main windows. But DriftFX doesn´t work on Linux, only on Windows. Which is the only reason for me to use imgui over DriftFX at this moment. |
and others
This is meant to be a meta-issue in which discussion can take place about potential solutions to GUI layout & rendering. Currently LWJGL only supports text rendering (via stb) and may soon support antialiased 2D graphics (via nanovg).
Topics to discuss:
The ideal solution should be hardware-accelerated, i.e. there shouldn't be any framebuffer copies from the CPU to the GPU or vice versa.
Approaches that have been explored already:
PixelWriter&PixelReader. See the LWJGL-FX project (uses LWJGL 2 but the same issues apply to LWJGL 3). Not hardware-accelerated.The text was updated successfully, but these errors were encountered: