X310/TwinRX - RFNoC: OpTimeout: Control operation timed out waiting for space in command buffer after using timed commands to set RX frequency.

[kkey12345]

Issue Description

Attempting to set TwinRX frequencies using a timed command to achieve phase coherence across multiple channels on multiple X310s. When using timed commands, UHD produces an error that the control operation timed out waiting for space in the command buffer. After this error, USRP is not usable until power-cycled.

Setup Details

UHD Version: 4.2.0.0-release
FPGA Image: g8daa80c
OS: Ubuntu 18.04
C++ Version: 7.5.0
Boost: 106501
SDR: 2 Ettus X310
Daughterboards: 2 TwinRX daughterboards in each X310
Connection: Ports 0 and 1 configured for 10GbE, connected with SFP+
RX LO Source: First channel of second USRP set to internal, second channel on TwinRX set to companion, signal is split externally and shared across all other channels.
Clock and PPS: External

Expected Behavior

Expect that frequencies of RX channels will be set at the same time across all channels of USRPs so that phase is coherent among channels, USRPs, and data collections. Not expecting command buffer to timeout.

Actual Behaviour

After executing the code for the first time, here is the terminal output:

Making USRP device with IP Addresses: addr0=192.168.230.2,second_addr0=192.168.240.2,addr1=192.168.231.2,second_addr1=192.168.241.2
[INFO] [UHD] linux; GNU C++ version 7.5.0; Boost_106501; UHD_4.2.0.0-release
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] No GPSDO found
[INFO] [X300] Radio 1x clock: 200 MHz
[INFO] [GPS] No GPSDO found
[INFO] [X300] Radio 1x clock: 200 MHz
[INFO] [MULTI_USRP]     1) catch time transition at pps edge
[INFO] [MULTI_USRP]     2) set times next pps (synchronously)
Antennas Set
LO Sources Set
RX Frequencies set synchronously
RX Gain set
[ERROR] [UHD] Exception caught in safe-call.
  in ~x300_adc_ctrl_impl
  at /usr/src/uhd/uhd-4.2.0.0/host/lib/usrp/x300/x300_adc_ctrl.cpp:124
this->send_ads62p48_reg(0x40); -> RfnocError: OpTimeout: Control operation timed out waiting for space in command buffer
[ERROR] [UHD] Exception caught in safe-call.
  in ~x300_dac_ctrl_impl
  at /usr/src/uhd/uhd-4.2.0.0/host/lib/usrp/x300/x300_dac_ctrl.cpp:59
_iface->write_spi(_slaveno, spi_config_t::EDGE_RISE, ((0x01) << 8) | (0xEF), 16); _iface->write_spi(_slaveno, spi_config_t::EDGE_RISE, ((0x02) << 8) | (0x1F), 16); -> RfnocError: OpTimeout: Control operation timed out waiting for space in command buffer
[ERROR] [UHD] Exception caught in safe-call.
  in ~x300_adc_ctrl_impl
  at /usr/src/uhd/uhd-4.2.0.0/host/lib/usrp/x300/x300_adc_ctrl.cpp:124
this->send_ads62p48_reg(0x40); -> RfnocError: OpTimeout: Control operation timed out waiting for space in command buffer
[ERROR] [UHD] Exception caught in safe-call.
  in ~x300_dac_ctrl_impl
  at /usr/src/uhd/uhd-4.2.0.0/host/lib/usrp/x300/x300_dac_ctrl.cpp:59
_iface->write_spi(_slaveno, spi_config_t::EDGE_RISE, ((0x01) << 8) | (0xEF), 16); _iface->write_spi(_slaveno, spi_config_t::EDGE_RISE, ((0x02) << 8) | (0x1F), 16); -> RfnocError: OpTimeout: Control operation timed out waiting for space in command buffer

After executing one additional time after that, the USRP is not usable until it has been power-cycled:

Making USRP device with IP Addresses: addr0=192.168.230.2,second_addr0=192.168.240.2,addr1=192.168.231.2,second_addr1=192.168.241.2
[INFO] [UHD] linux; GNU C++ version 7.5.0; Boost_106501; UHD_4.2.0.0-release
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [X300] Radio 1x clock: 200 MHz
[INFO] [X300] Radio 1x clock: 200 MHz
terminate called after throwing an instance of 'uhd::runtime_error[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: RfnocError: OpTimeout: Control operation timed out waiting for ACK
'
  what():  RuntimeError: Failure to create rfnoc_graph.
Aborted (core dumped)

Steps to reproduce the problem

Compile the following code (modeled from: https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD) and execute:

#include <uhd/usrp/multi_usrp.hpp>
#include <string>
#include <vector>
#include <thread>

int main () {

    //Add IP addresses for X310
    std::string ip_addr = "addr0=192.168.230.2,second_addr0=192.168.240.2,addr1=192.168.231.2,second_addr1=192.168.241.2";
	std::cout << "Making USRP device with IP Addresses: " << ip_addr << std::endl;
	uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make(ip_addr);

	// Set clock source
	usrp->set_clock_source("external");
	usrp->set_time_source("external");
	usrp->set_time_unknown_pps(uhd::time_spec_t(0.0));

    // Set subdev spec
	usrp->set_rx_subdev_spec(uhd::usrp::subdev_spec_t("A:0 A:1 B:0 B:1"), 0);
	usrp->set_rx_subdev_spec(uhd::usrp::subdev_spec_t("A:0 A:1 B:0 B:1"), 1);

    // Set antennas for TwinRX
    usrp->set_rx_antenna("RX1", 0);
    usrp->set_rx_antenna("RX2", 1);
	usrp->set_rx_antenna("RX1", 2);
	usrp->set_rx_antenna("RX2", 3);
	usrp->set_rx_antenna("RX1", 4);
	usrp->set_rx_antenna("RX2", 6);
	usrp->set_rx_antenna("RX1", 6);
	usrp->set_rx_antenna("RX2", 7);

    std::cout << "Antennas Set" << std::endl;

    // Set LO sharing sources
    usrp->set_rx_lo_source("external", uhd::usrp::multi_usrp::ALL_LOS, 0);
    usrp->set_rx_lo_source("external", uhd::usrp::multi_usrp::ALL_LOS, 1);
    usrp->set_rx_lo_source("external", uhd::usrp::multi_usrp::ALL_LOS, 2);
    usrp->set_rx_lo_source("external", uhd::usrp::multi_usrp::ALL_LOS, 3);
    usrp->set_rx_lo_source("internal", uhd::usrp::multi_usrp::ALL_LOS, 4);
    usrp->set_rx_lo_source("companion", uhd::usrp::multi_usrp::ALL_LOS, 5);
    usrp->set_rx_lo_source("external", uhd::usrp::multi_usrp::ALL_LOS, 6);
    usrp->set_rx_lo_source("external", uhd::usrp::multi_usrp::ALL_LOS, 7);

    std::cout << "LO Sources Set" << std::endl;

    // Set center frequency tune request
    uhd::tune_request_t tune_req(5.8e9);

    // Clear command time
    usrp->clear_command_time();
    // Set command time
    usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1));

    // Set frequencies
    usrp->set_rx_freq(tune_req, 0);
    usrp->set_rx_freq(tune_req, 1);
    usrp->set_rx_freq(tune_req, 2);
    usrp->set_rx_freq(tune_req, 3);
    usrp->set_rx_freq(tune_req, 4);
    usrp->set_rx_freq(tune_req, 5);
    usrp->set_rx_freq(tune_req, 6);
    usrp->set_rx_freq(tune_req, 7);

    // Sleep
    std::this_thread::sleep_for(std::chrono::milliseconds(110));
    usrp->clear_command_time();

    std::cout << "RX Frequencies set synchronously" << std::endl;

    //Set RX Rate
    usrp->set_rx_rate(100e6);

    std::cout << "RX Gain set" << std::endl;

    return 0;

}