rfnoc_radio_loopback with rx and tx from same block gives error: read property `scaling@OUTPUT_EDGE:0'

[TSlivede]

Issue Description

rfnoc_radio_loopback --rx-blockid '0/Radio#1' --tx-blockid '0/Radio#1' --duration 2

doesn't work and prints error:

[ERROR] [RFNOC::GRAPH::DETAIL] Adding edge 0/DDC#1:0 -> 0/DUC#1:0 without disabling property_propagation_active will lead to unresolvable graph!
Error: RfnocError: Adding edge without disabling property_propagation_active will lead to unresolvable graph!

If rx-blockid and tx-blockid are different, it works as expected (My real usecase is not the example. I am using all 4 rx-channels. The example just seems like the easiest way to reproduce the problem. Edit: After finding the workaround it seems that this problem is independent of my problem.).

As the error message mentioned that property_propagation could be disabled, I tried modifying the example. I made my own copy of rfnoc_radio_loopback.cpp, copied connect_through_blocks directly into it and changed the last graph->connect call to disable property propagation (I set skip_property_propagation to true).

This solves the unresolvable graph error, but leads to another error:

Error: RuntimeError: AccessError: Attempting to read property `scaling@OUTPUT_EDGE:0' before it was initialized!

This error appears to occur at the graph->commit call.

Setup Details

• USRP:
  • N310
  • UHD 4.0.0.0-0-g90ce6062 (installed via mender, downloaded with uhd_images_downloader from ettus)
  • FPGA-image: uhd_image_loader --args "type=n3xx,fpga=HG"
• host:
  • Ubuntu 20
  • UHD 4.0.0.0-154-gb061af4f

Expected Behavior

I expected to be able to use rx and tx from the same radio control block in loopback configuration (receive and directly send out again), at least when I disable property propagation.

Actual Behaviour

I got the errors mentioned above.

Steps to reproduce the problem

Modified rfnoc_radio_loopback code:

//
// Copyright 2016 Ettus Research LLC
// Copyright 2018 Ettus Research, a National Instruments Company
//
// SPDX-License-Identifier: GPL-3.0-or-later
//

// Example UHD/RFNoC application: Connect an rx radio to a tx radio and
// run a loopback.

#include <uhd/rfnoc/block_id.hpp>
#include <uhd/rfnoc/mb_controller.hpp>
#include <uhd/rfnoc/radio_control.hpp>
#include <uhd/rfnoc_graph.hpp>
#include <uhd/types/tune_request.hpp>
#include <uhd/utils/graph_utils.hpp>
#include <uhd/utils/math.hpp>
#include <uhd/utils/safe_main.hpp>
#include <boost/format.hpp>
#include <boost/program_options.hpp>
#include <chrono>
#include <csignal>
#include <iostream>
#include <thread>
#include <numeric>

namespace po = boost::program_options;
using uhd::rfnoc::radio_control;
using namespace std::chrono_literals;

using uhd::rfnoc::rfnoc_graph;
using uhd::rfnoc::block_id_t;
using uhd::rfnoc::graph_edge_t;

void my_connect_through_blocks(rfnoc_graph::sptr graph,
    const block_id_t src_blk,
    const size_t src_port,
    const block_id_t dst_blk,
    const size_t dst_port)
{
    // First, create a chain from the source block to a stream endpoint
    auto block_chain = get_block_chain(graph, src_blk, src_port, true);
    UHD_LOG_TRACE("GRAPH_UTILS", "Found source chain for " + src_blk.to_string());
    // See if dst_blk is in our block_chain already
    const bool dst_found = std::accumulate(block_chain.begin(),
        block_chain.end(),
        false,
        [dst_blk, dst_port](bool dst_found, const graph_edge_t edge) {
            // This is our "accumulator" function that checks if the current_blk's ID and
            // input port match what we're looking for
            return dst_found
                   || (dst_blk.to_string() == edge.dst_blockid
                       && dst_port == edge.dst_port);
        });
    // If our dst_blk is in the chain already, make sure its the last element and continue
    if (dst_found) {
        UHD_LOG_TRACE(
            "GRAPH_UTILS", "Found dst_blk (" + dst_blk.to_string() + ") in source chain");
        while (dst_blk.to_string() == block_chain.back().dst_blockid
               && dst_port == block_chain.back().dst_port) {
            UHD_LOG_TRACE("GRAPH_UTILS",
                boost::format(
                    "Last block (%s:%d) doesn't match dst_blk (%s:%d); removing.")
                    % block_chain.back().dst_blockid % block_chain.back().dst_port
                    % dst_blk.to_string() % dst_port);
            block_chain.pop_back();
        }
    } else {
        // If we hadn't found dst_blk, find it now, then merge the two chain
        auto dest_chain = get_block_chain(graph, dst_blk, dst_port, false);
        block_chain.insert(block_chain.end(), dest_chain.begin(), dest_chain.end());
        UHD_LOG_TRACE(
            "GRAPH_UTILS", "Found destination chain for " + dst_blk.to_string());
    }

    // Finally, make all of the connections in our chain.
    // If we have SEPs in the chain, find them and directly
    // call connect on the src and dst blocks since calling
    // connect on SEPs is invalid
    std::string src_to_sep_id;
    size_t src_to_sep_port         = 0;
    bool has_src_to_sep_connection = false;
    std::string sep_to_dst_id;
    size_t sep_to_dst_port         = 0;
    bool has_sep_to_dst_connection = false;

    for (auto edge : block_chain) {
        if (uhd::rfnoc::block_id_t(edge.dst_blockid).match(uhd::rfnoc::NODE_ID_SEP)) {
            has_src_to_sep_connection = true;
            src_to_sep_id             = edge.src_blockid;
            src_to_sep_port           = edge.src_port;
        } else if (uhd::rfnoc::block_id_t(edge.src_blockid)
                       .match(uhd::rfnoc::NODE_ID_SEP)) {
            has_sep_to_dst_connection = true;
            sep_to_dst_id             = edge.dst_blockid;
            sep_to_dst_port           = edge.dst_port;
        } else {
            graph->connect(
                edge.src_blockid, edge.src_port, edge.dst_blockid, edge.dst_port);
        }
    }
    if (has_src_to_sep_connection && has_sep_to_dst_connection) {
        graph->connect(src_to_sep_id, src_to_sep_port, sep_to_dst_id, sep_to_dst_port, true);
    } else if (has_src_to_sep_connection != has_sep_to_dst_connection) {
        throw uhd::runtime_error(
            "[graph_utils] Incomplete path. Only one SEP edge found.");
    }
}

/****************************************************************************
 * SIGINT handling
 ***************************************************************************/
static bool stop_signal_called = false;
void sig_int_handler(int)
{
    stop_signal_called = true;
}

/****************************************************************************
 * main
 ***************************************************************************/
int UHD_SAFE_MAIN(int argc, char* argv[])
{
    // variables to be set by po
    std::string args, rx_ant, tx_ant, rx_blockid, tx_blockid, ref, pps;
    size_t total_num_samps, spp, rx_chan, tx_chan;
    double rate, rx_freq, tx_freq, rx_gain, tx_gain, rx_bw, tx_bw, total_time, setup_time;
    bool rx_timestamps;

    // setup the program options
    po::options_description desc("Allowed options");
    // clang-format off
    desc.add_options()
        ("help", "help message")
        ("args", po::value<std::string>(&args)->default_value(""), "UHD device address args")
        ("spp", po::value<size_t>(&spp)->default_value(0), "Samples per packet (reduce for lower latency)")
        ("rx-freq", po::value<double>(&rx_freq)->default_value(0.0), "Rx RF center frequency in Hz")
        ("tx-freq", po::value<double>(&tx_freq)->default_value(0.0), "Tx RF center frequency in Hz")
        ("rx-gain", po::value<double>(&rx_gain)->default_value(0.0), "Rx RF center gain in Hz")
        ("tx-gain", po::value<double>(&tx_gain)->default_value(0.0), "Tx RF center gain in Hz")
        ("rx-ant", po::value<std::string>(&rx_ant), "Receive antenna selection")
        ("tx-ant", po::value<std::string>(&tx_ant), "Transmit antenna selection")
        ("rx-blockid", po::value<std::string>(&rx_blockid)->default_value("0/Radio#0"), "Receive radio block ID")
        ("tx-blockid", po::value<std::string>(&tx_blockid)->default_value("0/Radio#1"), "Transmit radio block ID")
        ("rx-chan", po::value<size_t>(&rx_chan)->default_value(0), "Channel index on receive radio")
        ("tx-chan", po::value<size_t>(&tx_chan)->default_value(0), "Channel index on transmit radio")
        ("rx-bw", po::value<double>(&rx_bw), "RX analog frontend filter bandwidth in Hz")
        ("tx-bw", po::value<double>(&tx_bw), "TX analog frontend filter bandwidth in Hz")
        ("rx-timestamps", po::value<bool>(&rx_timestamps)->default_value(false), "Set timestamps on RX")
        ("setup", po::value<double>(&setup_time)->default_value(0.1), "seconds of setup time")
        ("nsamps", po::value<size_t>(&total_num_samps)->default_value(0), "total number of samples to receive")
        ("rate", po::value<double>(&rate)->default_value(0.0), "Sampling rate")
        ("duration", po::value<double>(&total_time)->default_value(0), "total number of seconds to receive")
        ("int-n", "Tune USRP with integer-N tuning")
        ("ref", po::value<std::string>(&ref)->default_value("internal"), "clock reference (internal, external, mimo, gpsdo)")
        ("pps", po::value<std::string>(&pps)->default_value("internal"), "PPS source (internal, external, mimo, gpsdo)")
    ;
    // clang-format on
    po::variables_map vm;
    po::store(po::parse_command_line(argc, argv, desc), vm);
    po::notify(vm);

    // print the help message
    if (vm.count("help")) {
        std::cout << boost::format("RFNoC: Radio loopback test %s") % desc << std::endl;
        std::cout
            << std::endl
            << "This application streams data from one radio to another using RFNoC.\n"
            << std::endl;
        return ~0;
    }

    /************************************************************************
     * Create device and block controls
     ***********************************************************************/
    std::cout << std::endl;
    std::cout << boost::format("Creating the RFNoC graph with args: %s...") % args
              << std::endl;
    uhd::rfnoc::rfnoc_graph::sptr graph = uhd::rfnoc::rfnoc_graph::make(args);

    // Create handles for radio objects
    uhd::rfnoc::block_id_t rx_radio_ctrl_id(rx_blockid);
    uhd::rfnoc::block_id_t tx_radio_ctrl_id(tx_blockid);
    // This next line will fail if the radio is not actually available
    uhd::rfnoc::radio_control::sptr rx_radio_ctrl =
        graph->get_block<uhd::rfnoc::radio_control>(rx_radio_ctrl_id);
    uhd::rfnoc::radio_control::sptr tx_radio_ctrl =
        graph->get_block<uhd::rfnoc::radio_control>(tx_radio_ctrl_id);
    std::cout << "Using RX radio " << rx_radio_ctrl_id << ", channel " << rx_chan
              << std::endl;
    std::cout << "Using TX radio " << tx_radio_ctrl_id << ", channel " << tx_chan
              << std::endl;
    size_t rx_mb_idx = rx_radio_ctrl_id.get_device_no();

    /************************************************************************
     * Set up radio
     ***********************************************************************/
    // Connect the RX radio to the TX radio
    my_connect_through_blocks(
        graph, rx_radio_ctrl_id, rx_chan, tx_radio_ctrl_id, tx_chan);
    graph->commit();

    rx_radio_ctrl->enable_rx_timestamps(rx_timestamps, rx_chan);

    // Set time and clock reference
    if (vm.count("ref")) {
        // Lock mboard clocks
        for (size_t i = 0; i < graph->get_num_mboards(); ++i) {
            graph->get_mb_controller(i)->set_clock_source(ref);
        }
    }
    if (vm.count("pps")) {
        // Lock mboard clocks
        for (size_t i = 0; i < graph->get_num_mboards(); ++i) {
            graph->get_mb_controller(i)->set_time_source(pps);
        }
    }

    // set the sample rate
    if (rate <= 0.0) {
        rate = rx_radio_ctrl->get_rate();
    } else {
        std::cout << boost::format("Setting RX Rate: %f Msps...") % (rate / 1e6)
                  << std::endl;
        rate = rx_radio_ctrl->set_rate(rate);
        std::cout << boost::format("Actual RX Rate: %f Msps...") % (rate / 1e6)
                  << std::endl
                  << std::endl;
    }

    // set the center frequency
    if (vm.count("rx-freq")) {
        std::cout << boost::format("Setting RX Freq: %f MHz...") % (rx_freq / 1e6)
                  << std::endl;
        uhd::tune_request_t tune_request(rx_freq);
        if (vm.count("int-n")) {
            tune_request.args = uhd::device_addr_t("mode_n=integer");
        }
        rx_radio_ctrl->set_rx_frequency(rx_freq, rx_chan);
        std::cout << boost::format("Actual RX Freq: %f MHz...")
                         % (rx_radio_ctrl->get_rx_frequency(rx_chan) / 1e6)
                  << std::endl
                  << std::endl;
    }
    if (vm.count("tx-freq")) {
        std::cout << boost::format("Setting TX Freq: %f MHz...") % (tx_freq / 1e6)
                  << std::endl;
        uhd::tune_request_t tune_request(tx_freq);
        if (vm.count("int-n")) {
            tune_request.args = uhd::device_addr_t("mode_n=integer");
        }
        tx_radio_ctrl->set_tx_frequency(tx_freq, tx_chan);
        std::cout << boost::format("Actual TX Freq: %f MHz...")
                         % (tx_radio_ctrl->get_tx_frequency(tx_chan) / 1e6)
                  << std::endl
                  << std::endl;
    }

    // set the rf gain
    if (vm.count("rx-gain")) {
        std::cout << boost::format("Setting RX Gain: %f dB...") % rx_gain << std::endl;
        rx_radio_ctrl->set_rx_gain(rx_gain, rx_chan);
        std::cout << boost::format("Actual RX Gain: %f dB...")
                         % rx_radio_ctrl->get_rx_gain(rx_chan)
                  << std::endl
                  << std::endl;
    }
    if (vm.count("tx-gain")) {
        std::cout << boost::format("Setting TX Gain: %f dB...") % tx_gain << std::endl;
        tx_radio_ctrl->set_tx_gain(tx_gain, tx_chan);
        std::cout << boost::format("Actual TX Gain: %f dB...")
                         % tx_radio_ctrl->get_tx_gain(tx_chan)
                  << std::endl
                  << std::endl;
    }

    // set the IF filter bandwidth
    if (vm.count("rx-bw")) {
        std::cout << boost::format("Setting RX Bandwidth: %f MHz...") % (rx_bw / 1e6)
                  << std::endl;
        rx_radio_ctrl->set_rx_bandwidth(rx_bw, rx_chan);
        std::cout << boost::format("Actual RX Bandwidth: %f MHz...")
                         % (rx_radio_ctrl->get_rx_bandwidth(rx_chan) / 1e6)
                  << std::endl
                  << std::endl;
    }
    if (vm.count("tx-bw")) {
        std::cout << boost::format("Setting TX Bandwidth: %f MHz...") % (tx_bw / 1e6)
                  << std::endl;
        tx_radio_ctrl->set_tx_bandwidth(tx_bw, tx_chan);
        std::cout << boost::format("Actual TX Bandwidth: %f MHz...")
                         % (tx_radio_ctrl->get_tx_bandwidth(tx_chan) / 1e6)
                  << std::endl
                  << std::endl;
    }

    // set the antennas
    if (vm.count("rx-ant")) {
        rx_radio_ctrl->set_rx_antenna(rx_ant, rx_chan);
    }
    if (vm.count("tx-ant")) {
        tx_radio_ctrl->set_tx_antenna(tx_ant, tx_chan);
    }

    // check Ref and LO Lock detect
    if (not vm.count("skip-lo")) {
        // TODO
        // check_locked_sensor(usrp->get_rx_sensor_names(0), "lo_locked",
        // boost::bind(&uhd::usrp::multi_usrp::get_rx_sensor, usrp, _1, radio_id),
        // setup_time); if (ref == "external")
        // check_locked_sensor(usrp->get_mboard_sensor_names(0), "ref_locked",
        // boost::bind(&uhd::usrp::multi_usrp::get_mboard_sensor, usrp, _1, radio_id),
        // setup_time);
    }

    if (vm.count("spp")) {
        std::cout << "Setting samples per packet to: " << spp << std::endl;
        rx_radio_ctrl->set_property<int>("spp", spp, 0);
        spp = rx_radio_ctrl->get_property<int>("spp", 0);
        std::cout << "Actual samples per packet = " << spp << std::endl;
    }

    // Allow for some setup time
    std::this_thread::sleep_for(1s * setup_time);

    // Arm SIGINT handler
    std::signal(SIGINT, &sig_int_handler);

    // Calculate timeout and set timers
    // We just need to check is nsamps was set, otherwise we'll use the duration
    if (total_num_samps > 0) {
        total_time = total_num_samps / rate;
        std::cout << boost::format("Expected streaming time: %.3f") % total_time
                  << std::endl;
    }

    // Start streaming
    uhd::stream_cmd_t stream_cmd((total_num_samps == 0)
                                     ? uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS
                                     : uhd::stream_cmd_t::STREAM_MODE_NUM_SAMPS_AND_DONE);
    stream_cmd.num_samps  = size_t(total_num_samps);
    stream_cmd.stream_now = false;
    stream_cmd.time_spec =
        graph->get_mb_controller(rx_mb_idx)->get_timekeeper(rx_mb_idx)->get_time_now()
        + setup_time;
    std::cout << "Issuing start stream cmd..." << std::endl;
    rx_radio_ctrl->issue_stream_cmd(stream_cmd, rx_chan);
    std::cout << "Wait..." << std::endl;

    // Wait until we can exit
    uhd::time_spec_t elapsed_time = 0.0;
    while (not stop_signal_called) {
        std::this_thread::sleep_for(100ms);
        if (total_time > 0.0) {
            elapsed_time += 0.1;
            if (elapsed_time > total_time) {
                break;
            }
        }
    }

    // Stop radio
    stream_cmd.stream_mode = uhd::stream_cmd_t::STREAM_MODE_STOP_CONTINUOUS;
    std::cout << "Issuing stop stream cmd..." << std::endl;
    rx_radio_ctrl->issue_stream_cmd(stream_cmd, rx_chan);
    std::cout << "Done" << std::endl;
    // Allow for the samples and ACKs to propagate
    std::this_thread::sleep_for(100ms);

    return EXIT_SUCCESS;
}

Additional Information

Stacktrace of the location, where AccessError: Attempting to read ... before it was initialized! is thrown.

libuhd.so.4.1.0!uhd::rfnoc::property_t<double>::get(const uhd::rfnoc::property_t<double> * const this) (/home/user/ettus-project/uhd_debug_build/host/include/uhd/rfnoc/property.hpp:251)
libuhd.so.4.1.0!uhd::rfnoc::property_t<double>::equal(const uhd::rfnoc::property_t<double> * const this, uhd::rfnoc::property_base_t * rhs) (/home/user/ettus-project/uhd_debug_build/host/include/uhd/rfnoc/property.hpp:182)
libuhd.so.4.1.0!uhd::rfnoc::detail::graph_t::_assert_edge_props_consistent(uhd::rfnoc::detail::graph_t * const this, boost::adjacency_list<boost::vecS, boost::vecS, boost::bidirectionalS, boost::property<uhd::rfnoc::detail::graph_t::vertex_property_t, uhd::rfnoc::node_t*, boost::no_property>, boost::property<uhd::rfnoc::detail::graph_t::edge_property_t, uhd::rfnoc::graph_edge_t, boost::no_property>, boost::no_property, boost::listS>::edge_descriptor edge) (/home/user/ettus-project/uhd_debug_build/host/lib/rfnoc/graph.cpp:673)
libuhd.so.4.1.0!uhd::rfnoc::detail::graph_t::resolve_all_properties(uhd::rfnoc::detail::graph_t * const this, uhd::rfnoc::resolve_context context, boost::adjacency_list<boost::vecS, boost::vecS, boost::bidirectionalS, boost::property<uhd::rfnoc::detail::graph_t::vertex_property_t, uhd::rfnoc::node_t*, boost::no_property>, boost::property<uhd::rfnoc::detail::graph_t::edge_property_t, uhd::rfnoc::graph_edge_t, boost::no_property>, boost::no_property, boost::listS>::vertex_descriptor initial_node) (/usr/include/boost/graph/detail/edge.hpp:41)
libuhd.so.4.1.0!uhd::rfnoc::detail::graph_t::commit(uhd::rfnoc::detail::graph_t * const this) (/usr/include/boost/range/irange.hpp:89)
libuhd.so.4.1.0!rfnoc_graph_impl::commit(rfnoc_graph_impl * const this) (/usr/include/c++/9/bits/unique_ptr.h:360)
_main(int argc, char ** argv) (/home/user/ettus-project/devl/my-rfnoc-module/apps/rfnoc_radio_loopback_modified.cpp:201)
main(int argc, char ** argv) (/home/user/ettus-project/devl/my-rfnoc-module/apps/rfnoc_radio_loopback_modified.cpp:122)

Local variables of graph_t::_assert_edge_props_consistent(rfnoc_graph_t::edge_descriptor) then:

src_node->get_unique_id(): "0/DDC#0"
dst_node->get_unique_id(): "0/DUC#1"

[TSlivede]

I now have a workaround: I manually set the uninitialized scaling properties by hand. This is most certainly not the intended way, but it works.

To do this I copied node_accessor.hpp and prop_accessor.hpp to a place where they are accessible for my project (definitly not recommended...) and included them into my modified rfnoc_radio_loopback example. Then directly after the line I already modified (graph->connect(src_to_sep_id, src_to_sep_port, sep_to_dst_id, sep_to_dst_port, true);, 6th to last line in my_connect_through_blocks() in posted code above) I added:

        const uhd::rfnoc::res_source_info src_edge_info{uhd::rfnoc::res_source_info::OUTPUT_EDGE, src_to_sep_port};
        const uhd::rfnoc::res_source_info dst_edge_info{uhd::rfnoc::res_source_info::INPUT_EDGE, sep_to_dst_port};
        auto fix_scaling = [&](auto id, auto edge){
            auto props_ptrs = uhd::rfnoc::node_accessor_t{}.filter_props(
                graph->get_block(id).get(),
                [&](auto x) { return x->get_id() == "scaling"
                                  && x->get_src_info() == edge; });
            for (auto &&prop_ptr : props_ptrs)
            {
                auto prop = dynamic_cast<uhd::rfnoc::property_t<double> *>(prop_ptr);
                if(!prop->is_valid()){
                    uhd::rfnoc::prop_accessor_t{}.set_access(prop, uhd::rfnoc::property_base_t::RW);
                    prop->set(1.0);
                }
            }
        };
        fix_scaling(src_to_sep_id, src_edge_info);
        fix_scaling(sep_to_dst_id, dst_edge_info);

As I said: this works, but whatever the actual problem is should probably be properly fixed somewhere in the uhd library.

[andreaskuster]

@TSlivede
I ran into the same issue, but actually found a simpler work-round, which can be summarized as follow:

• switch from connect_through_blocks to the graph->connect function (all, even the static connections have to be setup manually)
• break the loop of property propagation, i.e. in my case, from my custom block to the DUC by setting skip_property_propagation=true

    /*
     * Connect blocks (blockname:port)
     */
    // radio0:0 -> ddc0:0
    graph->connect(rx_radio_id, rx_radio_chan, ddc_block_id, ddc_chan);
    // ddc0:0 -> custom0:0
    graph->connect(ddc_block_id, ddc_chan, custom_block_ctrl->get_block_id(), 0);
    // custom0:0 -> duc0:0
    graph->connect(custom_block_ctrl->get_block_id(), 1, duc_block_id, duc_chan, true);
    // duc0:0 -> radio0:0
    graph->connect(duc_block_id, duc_chan, tx_radio_id, tx_radio_chan);

    graph->commit();

Note that I did not test this example without my custom block in the chain, however, this should not change much.

[mbr0wn]

connect_through_blocks() now also has that argument, and that's the correct way to handle this.