EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error UHD 3.15.LTS and UHD 3.14

[mwk088]

Issue Description

X310 Failure after limited run. Full error below.

thread[thread-per-block[7]: <block gr uhd usrp source (3)>]: EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 3318 Received index: 3319

Setup Details

X310, two TwinRx daughterboards. Scanning between 2 frequencies every 11secs.
10Gig connection, two channel collect @100Msps, 80Mhz BW
UHD: 3.15.LTS, Also seen with UHD 3.14
OS: CentOS 7.9
Python 2.7.5
GNU: 3.7.14.0
Docker Environment
Server based system

Expected Behavior

Collection to continue and not error out

Actual Behaviour

Error message above seen and collection stops. I then must do a ctrl-C to fully exit out then restart my application again to retask the X310

Steps to reproduce the problem

Flowgraph includes UHD source block, with a number of demod blocks.

Additional Information

Numerous D's and O's seen leading up to the error. After performing a ctrl-C to completely exit the application, I get the error messages below.

[ERROR] [UHD] Exception caught in safe-call.
  in virtual uhd::rfnoc::x300_radio_ctrl_impl::~x300_radio_ctrl_impl()
  at /**/src/uhd/host/lib/usrp/x300/x300_radio_ctrl_impl.cpp:226
_tree->remove(fs_path("rx_codecs" / _radio_slot)); _tree->remove(fs_path("tx_codecs" / _radio_slot)); _tree->remove(_root_path / "rx_fe_corrections"); _tree->remove(_root_path / "tx_fe_corrections"); if (_radio_type == PRIMARY) { for (const gpio_atr::gpio_attr_map_t::value_type attr : gpio_atr::gpio_attr_map) { _tree->remove(fs_path("gpio") / "FP0" / attr.second); } } if (_radio_type == PRIMARY) { _regs->misc_outs_reg.set(radio_regmap_t::misc_outs_reg_t::ADC_RESET, 1); _regs->misc_outs_reg.set(radio_regmap_t::misc_outs_reg_t::DAC_RESET_N, 0); } _regs->misc_outs_reg.write(radio_regmap_t::misc_outs_reg_t::DAC_ENABLED, 0); _regs->misc_outs_reg.flush(); -> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 3320 Received index: 3321
[ERROR] [UHD] Exception caught in safe-call.
  in virtual dboard_manager_impl::~dboard_manager_impl()
  at /**/src/uhd/host/lib/usrp/dboard_manager.cpp:508
set_nice_dboard_if(); -> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 3321 Received index: 3322
[ERROR] [UHD] Exception caught in safe-call.
  in virtual twinrx_ctrl_impl::~twinrx_ctrl_impl()
  at /**/src/uhd/host/lib/usrp/dboard/twinrx/twinrx_ctrl.cpp:149
boost::lock_guard<boost::mutex> lock(_mutex); _gpio_iface->set_field(twinrx_gpio::FIELD_SWPS_EN, 0); -> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 3322 Received index: 3323
[ERROR] [UHD] Exception caught in safe-call.
  in adf535x_impl<adf535x_regs_t>::~adf535x_impl() [with adf535x_regs_t = adf5355_regs_t]
  at /**/src/uhd/host/lib/include/uhdlib/usrp/common/adf535x.hpp:132
_regs.power_down = adf535x_regs_t::POWER_DOWN_ENABLED; commit(); -> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 3323 Received index: 3324
[ERROR] [UHD] Exception caught in safe-call.
  in adf535x_impl<adf535x_regs_t>::~adf535x_impl() [with adf535x_regs_t = adf5355_regs_t]
  at /**/src/uhd/host/lib/include/uhdlib/usrp/common/adf535x.hpp:132
_regs.power_down = adf535x_regs_t::POWER_DOWN_ENABLED; commit(); -> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 3324 Received index: 3325
terminate called after throwing an instance of 'uhd::io_error'
  what():  EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 3325 Received index: 3326
Aborted (core dumped)

[mbr0wn]

Expected packet index: 3318 Received index: 3319

This means a control packet was lost somewhere (on the wire maybe).

[mbr0wn]

Can you please share some info about your application? Is this reproducible with UHD examples?

[mwk088]

Sure, my application is collecting wifi packets, demodulating them whether they are ofdm or Dsss and producing pcap files or running to null. My application creates the flowgraph like so...USRP -> Burst Detector -> Eventstream -> Demod blocks -> null.

I haven't run any UHD examples to try and reproduce. Are there any you suggest I use?

[mbr0wn]

twinrx_freq_hopping would be a good one.

[mbr0wn]

Cannot reproduce in this state.