Making measures clickable and styleable

[samuelbradshaw]

Thanks to pull request #2280, it's now possible to get the MIDI timestamp / milliseconds that corresponds to a specific measure in the sheet music.

I'd like to allow users in my web app to click or tap on a measure in the SVG sheet music, and use JavaScript to start MIDI playback at that time. However, in the SVG structure, measures are SVG group elements (<g>), and group elements don't respond directly to click events.*

Elements inside the group do respond to click events, and are able to pass the event to the measure group. Therefore, as a potential solution, would it be possible to add transparent <rect> elements inside each measure (behind the text and other elements) in the generated SVG?

An additional benefit to adding <rect> elements is that it would make it possible to add a background color to a measure (for example, on hover) by adjusting the fill of the <rect>.

(*In Chrome, it's possible to make group elements respond directly to click and hover events using pointer-events: bounding-box; however, this is not supported in other browsers.)

Related:
Stack Overflow: Add onClick event to a group element (svg) with react
Stack Overflow: CSS :hover on SVG group area instead of rendered pixels, pointer-events: bounding-box not working cross browser. How to workaround

[ahankinson]

If you add a rect as an element behind the other shapes, does that mean that clicking on a shape in front of it would not receive the click event?

[samuelbradshaw]

I haven't tested yet, but I would expect the front-most shape to receive the click event first. If the rectangle is behind the other shapes, it shouldn't prevent clicking shapes in front of it, selecting text, etc. By front, I mean the shape that is drawn closest to the user on the z-axis, or the shape that has the highest z-index.

[ahankinson]

I was actually thinking the other way -- if the click handler is on the rect and your user clicks a note, then they wouldn't jump to that measure.

[craigsapp]

If you add a rect as an element behind the other shapes, does that mean that clicking on a shape in front of it would not receive the click event?

Are you not supposed to be listening to papers right now? 😛

I would say yes (if the rect is placed first in the SVG image before the content of the measure, of course).

---

@samuelbradshaw I would be inclined to say that it should not be added to verovio, and should be added as post processing of SVG output from verovio. But this is open to debate, of course.

In particular, I want to be able to add analytic markup below (and perhaps above) the musical score. This can be demonstrated by a prototype of highlighting a measure from the URL anchor in VHV:

https://verovio.humdrum.org/?file=chorales/chor123.krn#mh8

You can example the SVG image to see what is happening.

Here is the Javascript code for doing this (with some generalization for later expansion of the demo):

https://github.com/humdrum-tools/verovio-humdrum-viewer/blob/gh-pages/_includes/vhv-scripts/highlight.js

So, I would not like a hard-coded system to do what you want to interfere with my markups. And related, you could adapt this code to insert the transparent rects in a similar manner yourself after the SVG is produced. Otherwise it is an application-specific use of the SVG output, so better in my opinion to not build into verovio. But it could be part of a more general analytic markup toolkit either inside verovio (via C++) or external post-processing of the SVG image in Javascript, as I am doing.

[samuelbradshaw]

The click handler would stay on the <g> element, and not on the <rect>. In my testing, I added a click listener to each <g> with the measure class. If I clicked on a note, lyric, or any other element inside the measure, the click listener fired for the <g>. If I clicked in the blank space between elements, nothing happened.

This video might help illustrate, using a hover event instead of a click event – the hover event is tied to the measure group. If I hover over any element inside the measure, the measure shows an outline. If I hover in the empty space, hover isn't detected, and the outline isn't drawn. The <rect> would fill in the empty space, allowing me to hover anywhere and trigger the measure's event listener.
https://user-images.githubusercontent.com/7323052/126524975-83fd4264-4ea0-4a25-b924-8af58e5750f7.mov

[ahankinson]

@craigsapp makes a good point; can you select all of the measure group elements using a DOM call and insert a rect in the DOM "on-the-fly"? That would solve your specific use case, and mean there wasn't a "workaround" for this specific use case baked in to the Verovio core.

[craigsapp]

The click handler would stay on the element, and not on the . In my testing, I added a click listener to each with the measure class. If I clicked on a note, lyric, or any other element inside the measure, the click listener fired for the . If I clicked in the blank space between elements, nothing happened.

Are you using event delegation? In other words one click event handler that examines the click target and acts on it rather than adding a separate click handler for every element -- this is for efficiency and scalability reasons, although maybe all of your examples may be small enough if there are less than 1000 elements in the svg perhaps.

https://stackoverflow.com/questions/1687296/what-is-dom-event-delegation

[samuelbradshaw]

So, I would not like a hard-coded system to do what you want to interfere with my markups.

I'll need to look in more detail later today. I wouldn't want to break your highlighting system, but I wouldn't expect that adding a transparent rectangle as a child of each measure would affect that.

Are you using event delegation? In other words one click event handler that examines the click target and acts on it rather than adding a separate click handler for every element -- this is for efficiency and scalability reasons, although maybe all of your examples may be small enough if there are less than 1000 elements in the svg perhaps.

Thanks for the link, I'll look at this as well later today. However, unless you've already verified that it works, I'm not too hopeful, since SVG <g> elements don't behave like HTML elements for targeting and responding to events (see the Stack Overflow links at the beginning of this thread).

[craigsapp]

One thing that would be good related to this issue is for verovio to have hard-coded (or optional) markup layers. This would be of benefit to both me and @samuelbradshaw. In side of the markup element we can add our own markup contents, such as the transparent rects for each measure.

Here is the markup system that I am usinging in the highlighter.js code:

<g class="page-margin" transform="translate(300, 1000)">
    <g class="lowerMarkupLayer">
        <g class="lowerSystemMarkups"></g>
        <g class="lowerMeasureMarkups">
            <rect x="11864" y="7739.0400390625" width="4361" height="7029.9599609375" fill="goldenrod" opacity="0.25">
            </rect>
        </g>
    </g>
...
</g>

The g.lowerMarkupLayer is inserted as the first child of g.page-margin

I also add two child elements that would probably also be useful to add (but is not necessary since these could be added as desired by application). The first one is for markup at the system level, then another one at the measure level. I will probably add another markup layer for note-level markup or sub-measure markup. In other words, I will use these three sub-element to control the basic z-index position of the markup elements.

-=+Craig

[samuelbradshaw]

Otherwise it is an application-specific use of the SVG output, so better in my opinion to not build into verovio.

I see it as more of a "application class"-specific use of the SVG output, meaning that any application that uses Verovio output in an interactive way – highlighting, clicking, playback, etc. – could benefit from it. :)