Child Windows under Wayland are clipped on resizing or on showing when placed partially outside the main window

[ggarra13]

Describe the bug
Wayland supports subwindows extending outside the boundaries of the main window.
However, FLTK clips and redraws them incorrectly when:

1. The subwindow is shown and it is partially outside its main parent window. It the subwindow is completely outside or completely inside the main window, there's no clipping.
2. The main window is resized and the subwindow is made to overlap. Leads to redraw issues.
3. The subwindow is resized and it is made to overlap the main window. Leads to redraw issues.

To Reproduce
See Issue #983 for a test program and for more information.

Expected behavior
Clipping should not happen as this is not a limitation of Wayland. Since Wayland's top windows have limited functionality it is imperative to support subwindows fully.

Linux/Unix Runtime, if applicable:

• Wayland

[ManoloFLTK]

The attached patch adds member function Fl_Window::can_leak_outside_parent(bool) to remove the clipping of subwindows. Does it fix the issue?

issue987.patch

[ggarra13]

Excellent work, Manolo! It works flawlessly! Thank you very mucho! I noticed you also changed it for macOS (Fl_cocoa.mm), but I did not test it. Let me know if you want me to test it there too.

[ManoloFLTK]

OK. I have committed a slightly different patch that is effective only under Wayland and has this public member function: Fl_Window::allow_expand_outside_parent().

[ManoloFLTK]

Gonzalo: when using your test program, I find that the "Drag me" window wobbles a lot when it gets dragged. Rather than looking in detail at your dragging code, I have prepared a test program that allows to drag a subwindow around with the mouse. That's a small modification of FLTK's subwindow program (attached patch). With this way of following the mouse, I see no wobbling at all.

subwindow.patch

[ggarra13]

I did not get to update to the latest code so that's why you may see wobbling (also I was using a timeout which is not ideal and might need different values depending on the speed of your machine).

I see what you did (adding init_sizes, which might be the ticket I was missing).

The reason I went with a timeout was that I found that dragging and resizing heavy windows full of widgets would sometimes give really bad numbers (like -2455567) from event_x_root()/event_y_root().

UPDATE: I just tried the subwindow example and it happens that for me too. The subwindow goes flying into space as soon as I drag it on my Ubuntu 22.04.4 LTS box with the Noble libnvidia-egl-wayland1 drivers.

UPDATE2: I tried it again with the Mesa egl drivers and also got the same out of screen coordinates.

[ggarra13]

I found the timeout was needed as it seems the compositor/window manager gets into a crazy loop when you modify it from FL_EVENT.

[ggarra13]

Here's an update to my code to with timeouts and init_sizes on the timeout. Tell me if you see wobbling on your side.

wayland_dragsubwindow_bug_v1.zip

[ManoloFLTK]

I am not going to work with your code. It's too complex. We need a small program focussed only on the problem under consideration, moving a child window. subwindow has this property.

I believe the proper solution to the wobbling is different and does not involve a timeout with an arbitrary delay value.

What do you see in terms of wobbling with the subwindow program, the last source code of FLTK and the mesa driver ?

[ggarra13]

Well, my code actually works on my machine. Here's what your subwindow code does on mine. I just added some printouts. I attach your patched source code and the log of the offsets I get:

subwindow.zip

[ggarra13]

What do you see in terms of wobbling with the subwindow program, the last source code of FLTK and the mesa driver ?

I get no wobbling, just a subwindow that flies out into space and disappears from the screen.

[ggarra13]

I attach the log for the same code running on X11, which obviously works, but clips at the edge of the window.

log.txt

[ManoloFLTK]

I hope the modified subwindow test app should be fixed with this patch:

issue987V2.patch

[ManoloFLTK]

and this other patch is simpler and should be good too:

issue987V3.patch

[ggarra13]

Yes. Seems to work fine in the simple demo. I am testing it with my code.

[ggarra13]

Okay. I tested with my code with V3 patch. Dragging works fine. However, resizing the subwindow does not work properly. In my program, it stretches the window outside the screen if I drag from either the top or from the left (dragging from the right, bottom or bottom+right corner works fine). If dragging too far, it will lead to (window outside the screen and this error):

Can't create Cairo surface with cairo_image_surface_create_for_data()

[ggarra13]

I will try modify subwindow.cxx tomorrow adding some additional code to reproduce the behavior I am seeing. Today it is kind of late.

[ggarra13]

Oh well... I did it anyway. The code does not use resizable() but uses custom handle() checks to check resizing on all margins.

subwindow.cxx.txt

[ManoloFLTK]

OK. There was a remaining missing bit in Wayland support of interactive subwindow resize. I have now committed the modified Wayland code to FLTK master. Your modified subwindow.cxx runs now well with the last but unpatched FLTK code. I would guess your app will run OK too with it.

It's good you've made me complete the FLTK support of interactive moving and resizing of subwindows under Wayland. No test app allows to check that, and I had not thought at all about exploring this feature.

[ggarra13]

@ManoloFLTK Thanks for being so thorough! You allowed me to take out all sort of hacks from my code and it indeed runs fine.

Since you are a perfectionist, I think I should mention a similar redraw issue when dragging from the left my main window with an OpenGL widget, and the docking bar in an Fl_Flex that is present on both X11 and Wayland (Windows and macOS seem fine and have no redraw issues).
I'm CCing @Albrecht-S, albeit I don't think it is a Fl_Flex issue.

See here how dragging from the right or bottom right is fine. But when dragging from the left there's flickering and the GL window expands beyond the dock window.

Grabacion.de.pantalla.desde.03-06-24.09.54.11.webm

I'll be opening a new issue for this as soon as I create a "simple" reproduceable example.

[ManoloFLTK]

The video above is fast, it's difficult to follow what happens. I see at worse the GL scene only transiently above the right window part. That seems acceptable to me for a interactive resizing operation.

My understanding is that issue can be closed now.

[ggarra13]

The video above is fast, it's difficult to follow what happens. I see at worse the GL scene only transiently above the right window part.

It is indeed in general transient. Only once in my tests the redraw left garbage.

My understanding is that issue can be closed now.

I am indeed closing the issue. If I manage to get a simple test, I'll post a new issue. I also understand you guys want to get 1.4 out the door, so I am sorry for being a pest with these late bugs.

[ggarra13]

It's good you've made me complete the FLTK support of interactive moving and resizing of subwindows under Wayland. No test app allows to check that, and I had not thought at all about exploring this feature.

I'm afraid your patch is still not fully working perfectly. I cannot reproduce it with subwindow.cxx from this thread, but in some windows in my program I am still getting wrong values when resizing from the left:

Grabacion.de.pantalla.desde.10-06-24.13.29.58.webm

[ManoloFLTK]

Is the problem that you stretch the window to the left from its left side and you see its left side jump to the right?

[ggarra13]

With this patch to Fl_Flex, the docking behavior under Wayland is fixed in my program. However I don't think this is a correct fix, but a work-around of a Wayland bug.

fl_flex_v1.patch.txt

[ManoloFLTK]

The issues I saw with shapeV2.cxx:
1. The GL scene is not drawn on my Ubuntu 24.04.4 LTS box with NVidia drivers (I attach a picture).

What happens if you type ctrl-+ ? What happens if you resize the main window ? What happens if you drag the subwindow at right to its left until it enters the main window ? Does the GL scene appear after these actions ?

What happens if you run FLTK's unchanged subwindow program: does the GL scene appear ?

What happens if you run FLTK's unchanged subwindow program with FLTK at commit 216ddd0: does the GL scene appear ?

[ManoloFLTK]

I am attaching movies (with explanations)

Sorry, that's useless for me. I understand nothing of all that.

[ManoloFLTK]

You are hard-coding the size of dock bar, instead of using Fl_Flex's layout() function.

OK. I see this point. I attach shapeV3.cxx.txt that fixes this

shapeV3.cxx.txt

[ManoloFLTK]

Is there a round trip under Wayland that needs to happen so that the GL Window updates its sizes, Manolo?

Test program shapeV3.cxx (or V2) tests resize operations inside the Fl_Flex, but not resizing the Fl_Flex itself inside the main window. There might be a missing thing in commit 1c6a0c1. Please, try applying the attached small patch to FLTK. It should render your fl_flex_v1.patch un-necessary.

resize-Fl_Flex.patch

[ggarra13]

The issues I saw with shapeV2.cxx:

1. The GL scene is not drawn on my Ubuntu 24.04.4 LTS box with NVidia drivers (I attach a picture).

What happens if you type ctrl-+ ?

The screen enlarges, the OpenGL does not show up.

What happens if you resize the main window ?

The black (GL?) window enlarges and the outside floating window follows along on the height, but not on the width or position.

What happens if you drag the subwindow at right to its left until it enters the main window ? Does the GL scene appear after these actions ?

Nope. The dock bar shows up, but the gl window does not appear.

What happens if you run FLTK's unchanged subwindow program: does the GL scene appear ?

What subwindow program with GL? Only shapeV2/3.cxx has a GL window.

What happens if you run FLTK's unchanged subwindow program with FLTK at commit 216ddd0: does the GL scene appear ?

It never opens a window and hangs. It just shows the icon on the dockbar.

[ggarra13]

Please, try applying the attached small patch to FLTK. It should render your fl_flex_v1.patch un-necessary.

resize-Fl_Flex.patch

The patch does not seem to apply cleanly. It complained with:

patching file FL/Fl_Group.H
patching file src/Fl_Flex.cxx
patch unexpectedly ends in middle of line

I tried it anyway and the x() position in the layout for the dock kept being incorrect.

[ManoloFLTK]

What happens if you run FLTK's unchanged subwindow program: does the GL scene appear ?

What subwindow program with GL? Only shapeV2/3.cxx has a GL window.

Sorry. I meant "What happens if you run FLTK's unchanged shape program: does the GL scene appear ?"

[ManoloFLTK]

I tried it anyway and the x() position in the layout for the dock kept being incorrect.

I just tried here to apply this patch to a clean, updated, FLTK source code. It applies cleanly with command patch -p1 < patch-file-name :

git pull
patch -p1 < /Users/mgouy/VBoxShared/linuxshared/fltk-macshared/resize-Fl_Flex.patch 
patching file 'FL/Fl_Group.H'
patching file 'src/Fl_Flex.cxx'

Could you, please, try again?

[ggarra13]

# gga@Ubuntu 24.04 LTS ~/code/lib/fltk (master)
# dom jun 30 04:19:37 [639] $ git switch master -f
Ya en 'master'
Tu rama está actualizada con 'origin/master'.

# gga@Ubuntu 24.04 LTS ~/code/lib/fltk (master)
# dom jun 30 04:19:45 [640] $ git pull
Ya está actualizado.

# gga@Ubuntu 24.04 LTS ~/code/lib/fltk (master)
# dom jun 30 04:19:53 [641] $ patch -p1 < resize-Fl_Flex.patch 
patching file FL/Fl_Group.H
patching file src/Fl_Flex.cxx
patch unexpectedly ends in middle of line

# gga@Ubuntu 24.04 LTS ~/code/lib/fltk (master)
# dom jun 30 04:19:54 [642] $ patch -v
GNU patch 2.7.6
Copyright (C) 2003, 2009-2012 Free Software Foundation, Inc.
Copyright (C) 1988 Larry Wall

[ggarra13]

Sorry. I meant "What happens if you run FLTK's unchanged shape program: does the GL scene appear ?"

No, it does not. But "cube" works fine.

[ManoloFLTK]

About resize-Fl_Flex.patch: there's a small mystery. Anyway, the important thing is to have 2 changes relatively to the FLTK of the git repo:

1. in file FL/Fl_Group.H, static int group_resize_depth_; must be in the protected instead of the private section of the class.
2. in file src/Fl_Flex.cxx, function Fl_Flex::resize() should become

void Fl_Flex::resize(int x, int y, int w, int h) {
  bool is_resize = (w != this->w() || h != this->h());
  if (is_resize) group_resize_depth_++;
  Fl_Widget::resize(x, y, w, h);
  layout();
  if (is_resize) group_resize_depth_--;
} // resize()

Sorry. I meant "What happens if you run FLTK's unchanged shape program: does the GL scene appear ?"
No, it does not. But "cube" works fine.

We have to keep separated two issues: how Fl_Flex resizes and how to map a GL subwindow with the NVidia driver.

1. please, try replacing the GL subwindow in shapeV3.cxx by a non-GL subwindow containing an Fl_Box with the label "GL here". Then run shapeV3 and report whether windows and subwindows and Fl_Flex behave correctly with resize-Fl_Flex.patch.

2. Can you find a way to map a GL subwindow correctly on screen? Your program does have several GL subwindow and they get mapped correctly. Are they all constantly changing as in cube? I think we had program shape working in the past with NVidia? I can't experiment here because I can't reproduce the problem.

[ggarra13]

We have to keep separated two issues: how Fl_Flex resizes and how to map a GL subwindow with the NVidia driver.

1. please, try replacing the GL subwindow in shapeV3.cxx by a non-GL subwindow containing an Fl_Box with the label "GL here". Then run shapeV3 and report whether windows and subwindows and Fl_Flex behave correctly with resize-Fl_Flex.patch.

The patch had the changes. Fl_Flex seems to behave properly in the shapeV3.cpp without a GL window. However, it does not behave correctly in my application. The GL window uses old widths when layout() is called. I noticed your patch did not modify anything in layout or GL windows that would change that behavior.

2. Can you find a way to map a GL subwindow correctly on screen?

OpenGL3test maps properly and does not have a constantly updating drawing. In shapeV3.cxx the window maps correctly (ie. clearing the GL screen with glClearColor does change the subwindow's color). It is the glVertex, etc. drawing calls that don't work. Could it be you are not setting compatibility with OpenGL's 2 core profile?

Your program does have several GL subwindow and they get mapped correctly. Are they all constantly changing as in cube?

Not really, but they are OpenGL3 and they use textures, like in cube and OpenGL3test.

I think we had program shape working in the past with NVidia? I can't experiment here because I can't reproduce the problem.

The NVidia problem was with EGL and that is fixed in Ubuntu 24.04.4 LTS.

[Albrecht-S]

@ManoloFLTK In comment #987 (comment) you wrote:

2. in file src/Fl_Flex.cxx, function Fl_Flex::resize() should become ...

This is a bad approach. Setting (incrementing) a static variable of Fl_Group inside a derived class and resetting (decrementing) it later is error prone: If this had to be done, then this would also apply to other subclasses derived by users and that's something we should really, really, really avoid.

Furthermore, this would not be an issue on all platforms except Wayland (if I understand the issue correctly), hence code developed on other platforms (by users and FLTK devs) would break when ported to or used on Wayland. That's IMO a no-go!

I noticed your recent commit when you introduced this variable and my gut feeling was "that's not how it should be done", but since it appeared to be a Wayland-only thing - and only in a core widget - I didn't interfere. But now this seems to become even worse.

There should be a better solution!

[ggarra13]

Sorry. I meant "What happens if you run FLTK's unchanged shape program: does the GL scene appear ?"

No, it does not. But "cube" works fine.

I am attaching a modified shape.cxx that actually works fine for me with NVidia drivers. The only change is that the drawing is done on the GL_BACK buffer. My actual program also has that modification for Wayland, as I noticed that drawing on the GL_FRONT buffer would not update until the next refresh. Not sure if it is an FLTK bug, NVidia bug or just the way Wayland works (I remember asking Google's Gemini that told me it was the way Wayland worked but I obviously don't trust it very much).

shape.cxx.txt

[ManoloFLTK]

The only change is that the drawing is done on the GL_BACK buffer. My actual program also has that modification for Wayland, as I noticed that drawing on the GL_FRONT buffer would not update until the next refresh.

That's very useful information, thanks. I may find a way out.

[ManoloFLTK]

About the new Fl_Group static variable counting the depth of group resize operations:

It was born because of a peculiar requirement of Wayland when moving or resizing subwindows: Wayland requires the window encompassing the resized or moved subwindow to be redrawn (in the Wayland sense, that is, to instruct the compositor to re-map its graphics content), in addition to the subwindow itself of course. This requirement is explicitly stated in the wl_subsurface chapter of the Wayland core protocol.

If such resize/move is done because some widget above the subwindow is resized, then, the encompassing window is redrawn anyway, so no additional redraw is needed. That's where the new static counter variable works: it gives a way to detect whether an additional operation is necessary.

Class Fl_Group has many derived classes. I've looked at all of them. Most call Fl_Group::resize() when they override their virtual resize() member. Two don't: Fl_Flex and Fl_Grid. Therefore, these two classes require to increment/decrement the new static counter. Others are unlikely to contain subwindows (e.g., Fl_Text_Display). Overall, it should be documented that when Fl_Group is derived and its resize() member is overridden by an FLTK user or developer, Fl_Group::resize() has to be called by the override or the counter has to be incremented/decremented. That's a situation for developers comparable, in my view, to the requirement to call Fl_Group::init_sizes() when children are rearranged.

This issue has arisen only very recently because subwindows had til now been resized only when their top-level window was. Until Gonzalo put forward the requirement for window docking, which in Wayland can be done with a subwindow that has to be moved or resized relatively to its non-resized parent window.

Another element of the issue is that Wayland requires to synchronize redraw operations with the compositor's ability to perform them. This happens when a (sub)window is resized or a subwindow is moved inside its parent. Wayland gives a way to check whether the compositor is ready for a new redraw of a particular (sub)window. When it's not, the redraw operation is not performed. Therefore, to move a subwindow requires to know whether its parent is ready for a redraw if it's not redrawn anyway by a previous operation.

Any other idea is welcome. As of today I've not found how to resize or move subwindows correctly under Wayland without the new counter variable.

[ManoloFLTK]

The only change is that the drawing is done on the GL_BACK buffer. My actual program also has that modification for Wayland, as I noticed that drawing on the GL_FRONT buffer would not update until the next refresh.

That's very useful information, thanks. I may find a way out.

@ggarra13 Here is a patch that hopefully could fix how the unchanged FLTK shape test app starts with the NVidia driver. Could you, please, try it and report here what you get?

fix-GL-map.patch.txt

[ggarra13]

@ggarra13 Here is a patch that hopefully could fix how the unchanged FLTK shape test app starts with the NVidia driver. Could you, please, try it and report here what you get?

Now I am getting black with or without glDrawBuffer(GL_BACK).

[ggarra13]

Now I am getting black with or without glDrawBuffer(GL_BACK).

Actually, I found I was mistaken and gave you wrong information. I sincerely apologize.

It is not glDrawBuffer(GL_BACK) that makes the NVidia driver work, but glReadBuffer(GL_BACK) at the end. I am attaching the working shape.cxx under Nvidia.

shape.cxx.txt

[ManoloFLTK]

It is not glDrawBuffer(GL_BACK) that makes the NVidia driver work, but glReadBuffer(GL_BACK) at the end.

Without access to a Linux box with an NVidia driver and without knowledge of OpenGl, I'm not able to solve this problem. Also, I'm very puzzled by the appearance of function glReadBuffer() to fix the shape test program because that function is documented as follows

glReadBuffer specifies a color buffer as the source for subsequent glReadPixels, glCopyTexImage1D, glCopyTexImage2D, glCopyTexSubImage1D, glCopyTexSubImage2D, and glCopyTexSubImage3D commands.

But none of these functions is used in the shape program!

If you replace in shape.cxx the statement glReadBuffer(GL_BACK); by

static int count=0; if(++count == 1) glReadBuffer(GL_BACK);

so that glReadBuffer() is called only once, does that repair the shape program? If so, I could modify the FLTK Wayland source code so it calls glReadBuffer(GL_BACK); once when creating a GL subwindow.

[ggarra13]

so that glReadBuffer() is called only once, does that repair the shape program?

It displays the triangle at the beginning, but as soon as you try the slider to change it to a circle, the display remains black. The call has to be permanent not just once.

But none of these functions is used in the shape program!

Right. But where/how does Wayland know where to write and copy the buffer from? And how does it inform Fl_Gl_Window about it? And viceversa.

Explanation about how OpenGL works on different OSes follows.

With OpenGL, under X11 and Windows, you can specify FL_DOUBLE or not.

If FL_DOUBLE is set there are two buffers: the back buffer and the front buffer. By default, with FLTK you draw to the back buffer and then FLTK is in charge of swapping the buffers by calling swap_buffers() once draw() returns.

Under macOS, even if FL_DOUBLE is not set, FLTK's behavior is that it creates two buffers nevertheless (I found that by trial and error). My understanding of Wayland (and from what I see now) is that it works like macOS too.

Under macOS and Wayland, it seems the default behavior, even without FL_DOUBLE, is to write to the back buffer that the Window manager creates. FLTK's macOS implementation automatically copies the back buffer to the front buffer once draw() returns.

Wayland seems like it is not qute doing that. With the glReadBuffer(GL_BACK) it seems like Wayland gets the hint to copy the back buffer it created with the NVidia drivers. Without it, it is either not triggering the copying or just copying the front buffer instead. This may be a bug in the NVidia driver or something missing on the FLTK implementation. You probably know better.

[Albrecht-S]

About the new Fl_Group static variable counting the depth of group resize operations ...

@ManoloFLTK I created GitHub Discussion #999 to separate this part of the issue from the main issue. I'm investigating and will follow-up with details there. This should be mostly independent of the main issue here.

[ManoloFLTK]

Wayland seems like it is not qute doing that. With the glReadBuffer(GL_BACK) it seems like Wayland gets the hint to copy the back buffer it created with the NVidia drivers. Without it, it is either not triggering the copying or just copying the front buffer instead. This may be a bug in the NVidia driver or something missing on the FLTK implementation.

The FLTK source code for Wayland adds the FL_DOUBLE flag to all Fl_Gl_Window's mode() if it's not present already. The net result is that all FLTK GL windows are double buffered under Wayland.

Could you, please, try to modify in file src/drivers/Wayland/Fl_Wayland_Gl_Window_Driver.cxx, the end of function Fl_Wayland_Gl_Window_Driver::swap_buffers() to obtain this

    }
    glReadBuffer(GL_BACK); // seems necessary with NVidia driver  <-- THIS LINE TO BE ADDED
    eglSwapBuffers(Fl_Wayland_Gl_Window_Driver::egl_display, egl_surface);
  }

and report whether FLTK and your GL-using apps work with NVidia with that. I have tried here this change and all GL apps tested that run well without it run also OK with it. If that makes apps work with NVidia under Wayland and if you tell me this change makes OpenGL sense, I will commit it.

You probably know better.

I don't because I understand nearly nothing of OpenGL.

[ggarra13]

Could you, please, try to modify in file src/drivers/Wayland/Fl_Wayland_Gl_Window_Driver.cxx, the end of function Fl_Wayland_Gl_Window_Driver::swap_buffers() to obtain this

That made the FLTK sample applications work. However, there's a lag of one redraw() with them. For example, dragging the Sides slider really fast on the shape demo (from 3 sides to circle and back), would leave it not as a triangle, but a square or polygon, depending on how fast I dragged.
I think we may need a glDrawBuffer(GL_BACK); also before draw() is called.

NOTE: The redraw() problem was also present when I added the glDrawBuffer(GL_BACK); on the main draw() function of shape.cxx.

[ggarra13]

I think the redraw issue is an NVidia bug. The best you can do is indeed keep the glReadBuffer(GL_BACK); which should be harmless anyway.

I want to recap where we are now:

1. Dragging, resizing and docking a subwindow with Fl_Flex works properly.
2. FLTK tests work properly on NVidia if you apply the patch and if we can live with one redraw behind.
3. With the subwindow change, there are now a lot of bugs regarding GL subwindows (I am opening a new issue).

If you are okay with 2), I am for closing this issue.

[ManoloFLTK]

1 Dragging, resizing and docking a subwindow with Fl_Flex works properly.
3 With the subwindow change, there are now a lot of bugs regarding GL subwindows (I am opening a new issue).

Points 1 and 3 above look contradictory to me. Does that mean that docking works OK only without any GL subwindow involved?

FLTK tests work properly on NVidia if you apply the patch and if we can live with one redraw behind.

I think it's not appropriate to apply this patch considering the following facts:

• an NVIdia driver bug seems to create the problem, not an error in FLTK ;
• the patch doesn't solve completely the GL drawing problem because some redraws remain missed;
• I don't understand why the patch would improve FLTK.

[ggarra13]

Points 1 and 3 above look contradictory to me. Does that mean that docking works OK only without any GL subwindow involved?

Docking of subwindows works properly now. But subwindows that don't have the leak outside flag and start in the main window do not resize/reposition themselves properly upon resizing of the main window. See the new issue #1003 I opened for a complete test suite of the problems that appear.

[ggarra13]

I think it's not appropriate to apply this patch considering the following facts:

Seems fine to me. The problem only shows in OpenGL 2 FLTK demos, not in OpenGL 3 code it seems.