Custom UI library

[slimsag]

We will not be using other prebuilt libraries (e.g. imgui) for Mach's UI framework. We will do something custom, which will be properly built on top of our ECS.

Short term goals:

• Built on top of our ECS, using mach.gfx.Sprite and mach.gfx.Text under the hood for rendering.
  • https://github.com/hexops/mach/tree/main/src/gfx
  • https://github.com/hexops/mach-examples/tree/main/examples/sprite
  • https://github.com/hexops/mach-examples/tree/main/examples/text (incomplete, see mach: text rendering ECS module #877)
  • mach: text rendering ECS module #877
• Usable for both games and regular desktop/mobile apps.
  • Could be used in browsers, only advised for games due to the lack of accessibility APIs browsers support.
• Modular; someone can create an ECS module which renders a UI (e.g. a module for rendering graphs) and others can reuse that.

Long term goals:

• gkurve rendering support, i.e. non-SVG vector graphics rendering
• Support WYSIWYG creation of UIs, via the general-purpose Mach editor and serialization format our ECS will mandate.
• Support basic/limited 3D UIs, such as what you would want in say VR games.
• Loads assets via our standard asset loading API.
• Native file open dialog
• General accessibility (we will get creative here) for regular non-game GUI desktop/mobile apps

[mateli]

For me personally this has a hard dependency on browser support.

My most relevant use case is that I need custom handling of enter keypress in a web form. This cannot as far as I know be done with anything that relies on the DOM to render HTML forms.

A TextBox component that allows for completely custom keyboard input processing the way that Windows.Forms or WPF does it is required.

Of course I do have other use cases for a browser UI solution that has it's own rendering and input handling. But I think that starting with something simple is important and a few basic UI components can get things done.

• Control - Things most component will share.
• Panel - A container for one or more control. This allows for building features like tabs, JInternalFrame etc. Although don't actually name a component "JInternalFrame". And don't split windowing functionality into two different classes (JFrame and JPanel).
• Label - Shows a simple text.
• TextBox - Allows for plain text editing
• Button - Usually useful to for example have a submit button.

As I have experience with a number of user interface toolkits I can provide a UI developers perspective on this. The first one would be to not heavily invest in having a large toolkit. It the basics are there third party custom controls will emerge.

In the early 1990's I managed to get my hands on Visual Basic which at the time were completely revolutionary when it came to building user interfaces. And its architecture are still visible in Windows.Forms, VCL, Xamarin/MAUI, AWT, Swing and Qt to mention a few.

However as the world are moving to the web many developers including me feel that browsers suck at this. Their rendering are slower than running a Windows 3.1 application inside an emulator, and placing components where you want them are hellish. While there are a number of JS libraries that simplifies this they can still not on the level of running Visual Basic 3. Not to mention 30+ years of further UI improvements.

This thread mentions accessibility features of modern browsers and they're a complete joke. More often than helping the user they become another obstacle.

To build a new UI that utilizes WebGPU is to me the only way forward to user friendly experience. Perhaps we can work together to upgrade web development to being competitive with Visual Basic 3.0. Beginning with implementing the three mentioned components.

Of course not all that exists in VB 3 were listed in my component list above. The reason for this is simple. Having the bare minimum amount of components rendering really well makes the UI library useful for a limited set of use cases. Having 200 bug infested components makes it Delphi.

Also while having a ful UI toolkit for both games and applications my reason for focusing on browsers are simple. This is where a new UI toolkit, even a very basic one, is most needed. Attracting web developers that want RAD is an easy way to establish market dominance because of the total lack of competition.

With web forms cornered a more extensive UI library can spread to other segment simply from the desire to reuse existing code. Writing user interfaces for both browsers and native applications in a single code base.

That is exactly how Java Swing became popular. You could write a good looking (although slow) user interface and use it both in applets and desktop applications. This is still a decent market as Red Hat still provides commercial support for it. This includes their own implementation of Java Applets that still have a team developing it. As well as extending their own fork of Firefox which ships with RHEL (Red Hat Enterprise Linus).

Now if Red Hat's paying customers could migrate to a full replacement of Swing/AWT that runs in an unmodified modern browser how much is that worth to them. Think about it. Right now anyone that needs to run such an applet must have RHEL as their desktop operating system. Meaning tablets and smartphones are not an option. Still enough commercial users are paying for this feature that RH has a complete team dedicated to it. Of course becoming a full replacement for Swing is not an easy task. But I bet that most of these users doesn't use even a fraction of all the components in Swing. They just use enough to be dissatisfied with the option of moving to inferior browser based solutions. Which we can solve for them.

So yeah that became quite a rant. But I wanted to provide a bit of background on why I think that focusing on browsers is so important.

Don't think games here. Think shopping chart forms. User registration forms. Database frontends. Inventory systems. Point of Sale terminals. Barcode scanners as a source of input.

[mateli]

Also while native open file dialogs are useful remember that on some platforms they're horrible. And in an enterprise environment the last thing you want is for users to notice that you switched from Windows to Linux by having to train them in how to use whatever native dialogs that the target OS is using.

Also when building full screen user interfaces one would usually want to stick to WebGPU as the only rendering pipeline that has any screen real estate.

Giving the developer a set of skinned native looking UI components are better. What if I want the OSX look and feel on all target platform? Well with a native-looking component in our toolkit they just set that look and feel. Or a native OS 9 look and feel. Or what about Windows Classic?

What I suggest is the following.

• Some native dialogs.
• Lightweight internal dialogs with VFS support.
• Multiple layouts that can mimic the differences between native toolkit. Even for platform we don't support (yet).
• Themes that can mimic the visual appearance of popular native toolkits.
• This should be obvious but DARK MODE support even when mimicking platforms that doesn't have a platform.

For further inspiration look at the Windows 95 similar for Android. Think about having a desktop component that can provide desktop functionality in full screen (similar to JDesktopPane) and then mimic everything that the user would expect on a desktop. Custom experience independent of what operating system the application is running on.