TEST/CALLBACKQ: Fix test_callbackq.test_callbackq test failures

[yosefe]

Why

Fix test failures like:

2024-07-10T10:31:07.4474989Z [ RUN      ] test_callbackq.oneshot_mt/mt_10 <> <>
2024-07-10T10:31:07.4478332Z /scrap/azure/agent-04/AZP_WORKSPACE/1/s/contrib/../test/gtest/ucs/test_callbackq.cc:360: Failure
2024-07-10T10:31:07.4478820Z Expected equality of these values:
2024-07-10T10:31:07.4479038Z   1u
2024-07-10T10:31:07.4479254Z     Which is: 1
2024-07-10T10:31:07.4479513Z   ctx.count
2024-07-10T10:31:07.4479810Z     Which is: 1
2024-07-10T10:31:07.4481242Z [  FAILED  ] test_callbackq.oneshot_mt/mt_10, where TypeParam =  and GetParam() =  (1 ms)
2024-07-10T10:31:07.4481612Z [ RUN      ] test_callbackq.remove_self_safe <> <>

[yosefe]

@tvegas1 @iyastreb can you pls review?

[iyastreb]

I counted 3 exiting reincarnations on exactly this function in UCX, and now this is the 4th.

ucp_test::wait_for_value
2 overloads in uct_test::wait_for_value

Apart from that there are multiple variations of the same function with slightly different names/behavior:

wait_progress
wait_for_rx_sn
many more...

Maybe we need to generalize it so that we don't have dozens of the similar functions?
Something like:

    template<typename Cond, typename Wait>
    void wait_for(Cond cond, Wait wait,
                        double timeout = DEFAULT_TIMEOUT_SEC) const
    {
        ucs_time_t deadline = ucs_get_time() + (ucs_time_from_sec(timeout) *
                                                ucs::test_time_multiplier());
        while ((ucs_get_time() < deadline) && (!cond())) {
            wait();
        }
    }

    template<typename T, Wait>
    void wait_for_value(volatile T *var, T value, Wait wait,
                        double timeout = DEFAULT_TIMEOUT_SEC) const
    {
        wait_for([&](){ return (*var == value); }, wait, timeout);
    }

Usage:
wait_for_value(..., sched_yield);

We can have overloads/default values for different wait strategies, or maybe leave it explicit

[yosefe]

fixed

[tvegas1]

maybe we can merge similar functions to some extent (wait_for_flag/wait_for_bits/wait_for_value..), gain might not be huge as code is small.

[yosefe]

/azp run UCX PR

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).