Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send sound to players which comes to hear distance. #14342

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

sfence
Copy link
Contributor

@sfence sfence commented Feb 3, 2024

To do

This PR is a Ready for Review.

How to test

Attach looped sound via Junkeox in devtest to object/location in distance bigger than heart distance and come to the area.

src/server.cpp Outdated Show resolved Hide resolved
src/server.cpp Outdated Show resolved Hide resolved
src/server.cpp Outdated Show resolved Hide resolved
@grorp grorp added the Action / change needed Code still needs changes (PR) / more information requested (Issues) label Feb 4, 2024
@Desour
Copy link
Member

Desour commented Feb 4, 2024

  • I'm a little worried about the O(player_count * sound_count) performance of this.
  • Entries from m_playing_sounds are deleted when all players have sent that they don't hear it anymore.
  • start_time should be used to make the playback offset the same for all players. E.g. if you enter an area where a sound has been playing for one minute, the sound should start at one minute.
  • This is a change in behavior. Are we sure nobody is relying on the old behavior? Anyway, needs change in doc, and a feature flag.
  • Mods can implement this equally well already (and they'll likely want to do other things as well, like completely fading out at distance). Is the change worth it? We could also just change the docs to recommend the use of per-player sounds instead of max hear distance if it's a long sound.

@sfence
Copy link
Contributor Author

sfence commented Feb 5, 2024

  • I'm a little worried about the O(player_count * sound_count) performance of this.

I do not know how many sounds are common there per player.
Do you have in mind some different solution for doing that check?

* Entries from `m_playing_sounds` are deleted when all players have sent that they don't hear it anymore.

So, I should not do this
https://github.com/minetest/minetest/blob/9ac12a2104714533a31f6967858b43267471084d/src/server.cpp#L2272C3-L2272C44
if the sound is not looped no one is in hear distance.
And sound should not be removed if is played in a loop and all clients are disconnected.

Sounds like a good solution?

* `start_time` should be used to make the playback offset the same for all players. E.g. if you enter an area where a sound has been playing for one minute, the sound should start at one minute.

Done.

* This is a change in behavior. Are we sure nobody is relying on the old behavior? Anyway, needs change in doc, and a feature flag.

* Mods can implement this equally well already (and they'll likely want to do other things as well, like completely fading out at distance). Is the change worth it? We could also just change the docs to recommend the use of per-player sounds instead of max hear distance if it's a long sound.

I will update doc and features.

@Desour
Copy link
Member

Desour commented Feb 5, 2024

I do not know how many sounds are common there per player.
Do you have in mind some different solution for doing that check?

You could at least do the iteration through the sounds in the outer loop, and early ignore non-max-hear-distance sounds.

* Entries from `m_playing_sounds` are deleted when all players have sent that they don't hear it anymore.

So, I should not do this https://github.com/minetest/minetest/blob/9ac12a2104714533a31f6967858b43267471084d/src/server.cpp#L2272C3-L2272C44 if the sound is not looped no one is in hear distance. And sound should not be removed if is played in a loop and all clients are disconnected.

Sounds like a good solution?

Link pasted by mistake?

If non-looped sounds don't appear if no player is near, I'd get very confused as modder and consider it as bug.
You need to remove the sound at some point. But we don't know when, because in lua they are just numbers and we don't know the sound length either.
I don't know of any simple solution for this.

This and the change in behavior would be enough conceptual issues for me to give up, tbh.

@sfence
Copy link
Contributor Author

sfence commented Feb 5, 2024

Question for core developers:

Situation:
Not-looped sound is added to the area with nobody in hearing distance. What do you think should happen?
Sound can be short and also long.

  1. Sound is added to m_playing_sounds and kept here until call core.sound_stop or until some player reaches the area and plays the sound.
  2. minetest.sound_play returns 0 to inform that the adding sound failed. It is on modder to deal with it.
  3. Somehow determine the max length of sound from file/files or some Lua table field. If in a specified time nobody reached a sound distance, remove the sound.
  4. Any other ideas?

@SmallJoker
Copy link
Member

SmallJoker commented Feb 5, 2024

Question for core developers:

The server does currently not have any media library dependencies (PNG, OGG), and I think that's OK for simplicity. Option 2 would IMO be the most reasonable approach because the sound could not be played to anyone.

src/server.cpp Outdated Show resolved Hide resolved
src/server.cpp Outdated Show resolved Hide resolved
src/server.cpp Outdated Show resolved Hide resolved
src/server.cpp Show resolved Hide resolved
src/server.cpp Show resolved Hide resolved
@SmallJoker
Copy link
Member

SmallJoker commented Feb 6, 2024

One more thought:

Imagine 2 players. One with low latency (< 10 ms), another with more than a server tick.
The high latency player will start the sound later and hence mark it as "done" later. If the delay is > 1 server tick, the server will re-send a packet to the low latency player (with a time offset past the sound length).

To reduce this issue, adding an additional time offset of the player's RTT could be considered. A better solution would be to no longer play back sounds after the first "done" packet is received.
EDIT: latter would require checking what happens when a client has no sound support - is the "done" packet sent immediately, or not at all?

@sfence
Copy link
Contributor Author

sfence commented Feb 8, 2024

@SmallJoker
I have tried it with minetest client build without sound. It answer with "done" packet pretty immediately.

There will probably have to be added some field to sound parameters to say, how long should sound wait before allow removeal.

@appgurueu appgurueu added the Rebase needed The PR needs to be rebased by its author. label Feb 25, 2024
@appgurueu
Copy link
Contributor

Please rebase. Now that #14341 is merged, I will dedicate some time to reviewing this, so that we can properly fix this.

@Zughy Zughy removed Rebase needed The PR needs to be rebased by its author. Action / change needed Code still needs changes (PR) / more information requested (Issues) labels Feb 27, 2024
@sfan5 sfan5 added the Rebase needed The PR needs to be rebased by its author. label Mar 19, 2024
@sfence sfence force-pushed the sfence_fix_3559 branch 2 times, most recently from 806c14f to 72f9d3b Compare April 5, 2024 14:24
@sfan5 sfan5 removed the Rebase needed The PR needs to be rebased by its author. label Apr 5, 2024
Copy link
Member

@SmallJoker SmallJoker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grammar, documentation.
I did yet not have the chance to test the PR's functionality but overall the code seems to make sense.

src/sound.h Outdated Show resolved Hide resolved
src/server.h Show resolved Hide resolved
doc/lua_api.md Outdated Show resolved Hide resolved
@@ -5435,6 +5440,9 @@ Utilities
-- Allow passing an optional "actor" ObjectRef to the following functions:
-- minetest.place_node, minetest.dig_node, minetest.punch_node (5.9.0)
node_interaction_actor = true,
-- Sounds can be send to players which comes to hear distance.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
-- Sounds can be send to players which comes to hear distance.
-- Sounds can be sent to players who come within hearing distance.

src/network/serverpackethandler.cpp Outdated Show resolved Hide resolved
src/server.cpp Outdated Show resolved Hide resolved
src/server.cpp Outdated Show resolved Hide resolved
src/server.cpp Outdated Show resolved Hide resolved
src/network/serverpackethandler.cpp Outdated Show resolved Hide resolved
doc/lua_api.md Show resolved Hide resolved
@Desour
Copy link
Member

Desour commented Apr 6, 2024

    max_hear_distance = 32,
-    -- Only play for players that are at most this far away when the sound
-    -- starts playing.
+    -- Only play for players that are at most this far away.
+    -- bla bla bla, see `sounds_updating`

Please don't return 0 to indicate failure. If mods compare handles to 0, we can't make the handles to userdata anymore as easily.

Edit:
Btw. please make sure to test what happens if the only player who hears a sound leaves the server.

if (pos.getDistanceFrom(plrsao->getBasePosition()) <= psound.max_hear_distance) {
/* If client remove sound in hear distance,
we are sure, that sound has been fully played. */
psound.done_clients.insert(peer_id);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic seems brittle to me: The positions on server and client may not be perfectly in sync, for example due to attached sounds, meaning that this check can't accurately match what the client is doing.

The straightforward fix might be to have the client additionally tell us why a sound was removed (whether it was removed for going out of range or for reaching the end of playback) by extending the packet. For older clients, we could still keep this hack as a decent approximation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no packet sent when the client comes out of the sound range. The Sound continues playing.
Clients generate stop sound packets only if the playing sound is finished on the client side. When sound is removed due to server request, no stop sound packet is generated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More testing is needed to remove this condition because I see unexpected changes in behaviors after removing it.

BTW: Before #14341, sound removal after object delete was not processed on the server side and there was a packet sent from client to server.

src/server.h Outdated
// used for update spec.start_time when sending sound to new client
float start_time;
// hold time limit for sound sending to new clients
float keep_time;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am slightly worried about this being a float due to potential precision issues (quick back of the napkin math suggests after 2^24 / 60 / 60 / 24 / 30, so approximately seven months, you will lose a second of precision). I'd bump this (and m_playing_sound_time ) to a double, or perhaps even a std::chrono::duration just to be sure. I'm also not sure how accurate dtime is, so I would consider using the system clock via std::chrono to get more accurate timestamps. Thoughts?

(Side note: This is tangentially related to the more general question of how we want to synchronize client and server.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, I would suggest the use of m_game_time and m_game_time_fraction_counter from the ServerEnvironment class.

Does it look fine for you?

@Desour
Copy link
Member

Desour commented Apr 6, 2024

It changes behaviour. Is this fine?

Does it? Given a keep_time parameter defaulting to 0, the default should be the old / "buggy" behavior, but modders should be able to opt in to the new behavior, or am I missing something?

Oh, I haven't looked well enough what keep_time does exactly, and misunderstood it.

sound_play being able to fail makes the API overall harder to use.

Is this really new?

What is new is that it now makes a difference if it fails. In master, you don't have to check if it failed, because it makes no difference if you call API functions with -1, and you don't have any effects if a sound exists server side or not.

Because sound_play fails if nobody is near, mods will still have to manually keep track where there should be a sound, otherwise they'll get bugs in rare situations.

If I understand you correctly, you have a problem with the fact that if there is no one near initially, sounds don't get sent to players coming into range later on. This indeed looks like a bug in this PR to me. I think if keep_time > 0, the sound should be added anyways, to let the server keep track of it.

If keep_time works like this, this PR can work, I think.

Maybe rename keep_time to something like resend_window (allowing math.huge, e.g. for looped sounds), and only resend in this time interval, no matter what the clients send or if it's looped. This way, we'd also gain perfect backwards compatibility. (Is it already intended to be like this?)

Using max_hear_distance for long sounds was a bad feature to begin with, because how it works in master. [...] We could just change the docs to advise against using max_hear_distance with long or looped sounds, instead of all this mess.

What do you want modders to use instead?

They can manually send the sound to the nearby players, and manage it themselves.
(This way, they can also fade out sounds if players leave the area.)

If you have to keep track of the sounds manually anyways, in what situations does this feature make the modder's life easier?

I'm a bit confused as to why you're speaking of a feature when this PR should be a bug fix. Do you not consider #3559 a bug, or do you think that this PR does not fix #3559 (and instead implements an unrelated feature)?

(I consider #3559 a missing feature, not a bug.)
If sound_play doesn't fail, if keep_time is given, my point is invalid.

@sfence
Copy link
Contributor Author

sfence commented Apr 8, 2024

@appgurueu @Desour
The main reason to add keep_time was testing with a client compiled without sound support. (#14342 (comment))

If you play a long sound without keep_time, and in the hearing distance is only the player playing on the client without sound, the client responds immediately with a sound "done" packet and the server removes the sound (the mod author has no way to detect this}. If later somebody who has a client with sounds comes into the hearing distance, nothing happens and it will be probably reported as a bug.

To prevent this, I added keep_time. For that time, sounds cannot be removed on the server side and will be waiting if somebody comes into hearing distance. The server cannot check how long the sounds are, from what I know, we do not want to make the server depend on the same multimedia libraries. There is no universal valid default keep_time value without knowing the length of sounds on the server side. So I selected 0. Only mod authors know how long sounds are. So, they should choose the best value.

So from my side of view keep_time can be removed if the client builds without sounds will be removed.

Or do you have a different idea of how to fix this?

doc/lua_api.md Outdated
@@ -1099,6 +1099,11 @@ Table used to specify how a sound is played:

-- Available since feature `sound_params_start_time`.

keep_time = 0.0,
-- Define time window which allow sound to be sent to players
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a good suggestion but it feels like we're trading one hack¹ for another by adding this parameter.
Maybe the server should just track all sounds for a duration of 10 minutes and automatically re-send them? It would be up to the client to skip sounds where start_time >= sound_length

¹: max_hear_distance is the previous hack

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used 600s as the default keep_time value in the last version.

@sfence
Copy link
Contributor Author

sfence commented Apr 19, 2024

@Desour

    max_hear_distance = 32,
-    -- Only play for players that are at most this far away when the sound
-    -- starts playing.
+    -- Only play for players that are at most this far away.
+    -- bla bla bla, see `sounds_updating`

Please don't return 0 to indicate failure. If mods compare handles to 0, we can't make the handles to userdata anymore as easily.

Edit: Btw. please make sure to test what happens if the only player who hears a sound leaves the server.

Please see: #14342 (comment)
and answer here #14342 (comment) .

Maybe it needs to be discussed more between core developers to find a good solution with the consensus.

@sfence
Copy link
Contributor Author

sfence commented Apr 20, 2024

I'm still having conceptual concerns about this change.

* It changes behaviour. Is this fine?

* `sound_play` being able to fail makes the API overall harder to use.

* Because `sound_play` fails if nobody is near, mods will still have to manually keep track where there should be a sound, otherwise they'll get bugs in rare situations. (And when testing their mod, modders will usually not encounter this issue, because their player is near the sound source.) If you have to keep track of the sounds manually anyways, in what situations does this feature make the modder's life easier? I can't think of an example where it would.

* Using `max_hear_distance` for long sounds was a bad feature to begin with, because how it works in master. And this PR has to rely on nasty things like clients telling the server when to remove a sound.
  We could just change the docs to advise against using `max_hear_distance` with long or looped sounds, instead of all this mess.

See #14342 (comment),

No problem with changing this behavior. But I have to know what core developers want. Now there is one core developer vote for returning failure and one vote for adding sound to sounds for at least keep_time is reached.

@Desour
Copy link
Member

Desour commented Apr 28, 2024

So from my side of view keep_time can be removed if the client builds without sounds will be removed.

Note that this is barely possible. If a server doesn't have a sound, the client can still get it from the user's sound pack. The server never sees the sound in this case.


Or do you have a different idea of how to fix this?

This here.
@sfence What are your (and other peoples' reading this) thoughts on this proposal?

Maybe rename keep_time to something like resend_window (allowing math.huge, e.g. for looped sounds), and only resend in this time interval, no matter what the clients send or if it's looped. This way, we'd also gain perfect backwards compatibility. (Is it already intended to be like this?)

Pros/Goals:

  • If nothing is provided (aka default 0), it's the old behaviour: Sound is not kept track of for sending to players that come into hear distance.
  • You have an easy way to tell how long the sound lives server-side, independent of the actual sound file. Also no special case for looped sounds.
    (Or actually, how long it lives for this feature. The sound can be kept track of twice. Once like in master, and then again (in another data structure) for this feature. When the time runs out you can remove the sound from this 2nd data structure. => simple logic and code)
  • If no player is near, the sound is still kept track of server-side, no need to return 0.
  • It's clear for modders that if they choose an upper bound of the sound length, it will always work.
  • If you allow inf, modders can use this and manually stop the sound with core.sound_stop. => No need to know sound length if it's something like a music box.

Idk if my gibberish makes sense to you.

@sfence
Copy link
Contributor Author

sfence commented May 1, 2024

So from my side of view keep_time can be removed if the client builds without sounds will be removed.

Note that this is barely possible. If a server doesn't have a sound, the client can still get it from the user's sound pack. The server never sees the sound in this case.
It is not about the user's sound pack, but about the client responding with a sound-done packet immediately if the client is compiled without sound support. But yes, the user's sound pack with sounds of zero length will be used, it behaves in the same way.

Or do you have a different idea of how to fix this?

This here. @sfence What are your (and other peoples' reading this) thoughts on this proposal?

Maybe rename keep_time to something like resend_window (allowing math.huge, e.g. for looped sounds), and only resend in this time interval, no matter what the clients send or if it's looped. This way, we'd also gain perfect backwards compatibility. (Is it already intended to be like this?)
I have no problem with renaming. But the default value 0 for keep_time has been criticized because it causes default behaviors to be still buggy (#3559).

Pros/Goals:

* If nothing is provided (aka default 0), it's the old behaviour: Sound is not kept track of for sending to players that come into hear distance.

Don't look like a supported solution, because without some actions, bug #3559 does not disappear.

* You have an easy way to tell how long the sound lives server-side, independent of the actual sound file. Also no special case for looped sounds.
  (Or actually, how long it lives for this feature. The sound can be kept track of twice. Once like in master, and then again (in another data structure) for this feature. When the time runs out you can remove the sound from this 2nd data structure. => simple logic and code)

* If no player is near, the sound is still kept track of server-side, no need to return 0.

I prefer this solution, but there are also opposite opinions.

* It's clear for modders that if they choose an upper bound of the sound length, it will always work.

* If you allow inf, modders can use this and manually stop the sound with `core.sound_stop`. => No need to know sound length if it's something like a music box.

What is the expected situation when this can be used?

Idk if my gibberish makes sense to you.

@Desour
Copy link
Member

Desour commented May 1, 2024

  • If you allow inf, modders can use this and manually stop the sound with core.sound_stop. => No need to know sound length if it's something like a music box.

What is the expected situation when this can be used?

If the mod uses inf, the server would never itself drop the sound. It will live until you call remove_sound.

(We could also have an API, where only the values 0 and inf are allowed (or false and true). Instead of providing a time limit, the modder can user core.after.)

@sfence
Copy link
Contributor Author

sfence commented May 1, 2024

  • If you allow inf, modders can use this and manually stop the sound with core.sound_stop. => No need to know sound length if it's something like a music box.

What is the expected situation when this can be used?

If the mod uses inf, the server would never itself drop the sound. It will live until you call remove_sound.

(We could also have an API, where only the values 0 and inf are allowed (or false and true). Instead of providing a time limit, the modder can user core.after.)

The looped sound behaves like inf. Is it useful for non-looped sounds to be unremovable automatically (After some time passes, the server only sends to a new client a start sound packet, but nothing will be played)?

@Desour
Copy link
Member

Desour commented May 1, 2024

Is it useful for non-looped sounds to be unremovable automatically (After some time passes, the server only sends to a new client a start sound packet, but nothing will be played)?

I've seen mods that add a music player where the sound is not looped. With un-automatically-removable sounds, you could just stop the sound when the music player is stopped, or it's unloaded. The modder wouldn't have to take the sound length into account.

@sfence
Copy link
Contributor Author

sfence commented May 1, 2024

Is it useful for non-looped sounds to be unremovable automatically (After some time passes, the server only sends to a new client a start sound packet, but nothing will be played)?

I've seen mods that add a music player where the sound is not looped. With un-automatically-removable sounds, you could just stop the sound when the music player is stopped, or it's unloaded. The modder wouldn't have to take the sound length into account.

You can stop every sound manually if you do not start it as ephemeral. But this mod probably counts on automatically removing sound after it is fully played. But yes, it makes sense to support Inf here, but probably with added stop sound reason to packet, to determine that sound has been fully played.

With that information in the packet, it can be possible to use keep_time with Inf default value and keep it as Inf until some client fully plays it and enables it as removable after it.

@grorp grorp added this to the 5.9.0 milestone May 24, 2024
@grorp
Copy link
Member

grorp commented May 24, 2024

(added to the milestone since #14341 and #14436 were merged for 5.9.0, so we have a new bug otherwise)

@grorp
Copy link
Member

grorp commented May 24, 2024

Re #14342 (comment):

Like appgurueu and Desour seem to do, I support the "Don't let minetest.sound_play fail if nobody is in hear distance. Add the sound anyway and play it if someone comes into hear distance, obviously with an appropriate time offset." side in this discussion.

Re the "nothing changes by default proposal" in #14342 (comment):

I think we need to resend sounds attached to objects when they come back into hear distance without any modder intervention. Otherwise the behavior introduced by #14436 doesn't make any sense and becomes a breaking change by itself.

@Zughy Zughy added the Action / change needed Code still needs changes (PR) / more information requested (Issues) label May 24, 2024
@Zughy Zughy added the Rebase needed The PR needs to be rebased by its author. label Jun 2, 2024
@Zughy
Copy link
Member

Zughy commented Jun 16, 2024

hey @sfence , any updates?

@appgurueu appgurueu removed the Rebase needed The PR needs to be rebased by its author. label Jun 16, 2024
@appgurueu appgurueu force-pushed the sfence_fix_3559 branch 2 times, most recently from c71c9de to 07738c4 Compare June 16, 2024 19:00
@sfence
Copy link
Contributor Author

sfence commented Jun 19, 2024

@Zughy I leave this to get more feedback.
I will look at it as soon as possible.

@appgurueu Why merge 'master' to this branch instead of rebase?

@sfence
Copy link
Contributor Author

sfence commented Jun 23, 2024

keep_time renamed to resend_time (reflection of #14342 (comment) )
The default value of resend_time is 0 -> as default, resending is active only for looped sounds.

@grorp
Copy link
Member

grorp commented Jun 27, 2024

I think this change

The default value of resend_time is 0 -> as default, resending is active only for looped sounds.

contradicts this request

I think we need to resend sounds attached to objects when they come back into hear distance without any modder intervention. Otherwise the behavior introduced by #14436 doesn't make any sense and becomes a breaking change by itself.

for non-looped sounds attached to objects.

@sfence
Copy link
Contributor Author

sfence commented Jun 28, 2024

@grorp I changed the default value of resend_time back to 600. I don't know if there is a better solution.

@Zughy Zughy removed the Action / change needed Code still needs changes (PR) / more information requested (Issues) label Jun 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Positional sound not playing if you walk into range
7 participants