X310: Incorrect mapping of channels in multi-USRP TX only setup (DmaFIFO-related)

[ab-kbg]

Issue Description

When using DmaFIFO (which is the default), the enabled channels are different than the intended ones, and change between each run of the program. In effect, Multi-USRP TX does not work properly with two or more X310s.

Setup Details

UHD 3.15.0.0 on Fedora 32 Linux
2 X310s motherboards, 192.168.10.2 and 192.168.110.2
4 UBX-40 daughterboards
Laptop with IPs 192.168.10.1 and 192.168.110.1, connected to the X310s via 1Gbit ethernet switch.
OctoClock-G connected to both X310s, providing 1 PPS and 10 MHz (not required to reproduce)

Expected Behavior

When I specify tx_channels="0,2" (subdevice spec is "A:0 B:0" by default), i should expect that the A output on both X310 gets enabled, while B outputs remain disabled.

Actual Behaviour

Sometimes, A and B outputs on first X310 are the only ones enabled. Other times A and B outputs on the second X310 are the only ones enabled. Rarely, <10% of the time, the behavior is as expected, where A outputs on both X310s are only ones enabled. The proportion between these errors changes somewhat depending on whether I enable external sync/ref or not, but external sync/ref is not required to demonstrate the issue.
This varies from run to run, and also occurs with other channel selections than "0,2".
When DmaFIFO is disabled via "skip_dram=1", the mapping is as expected, and the issue does not occur.

Steps to reproduce the problem

Both of these exhibit the issue:
benchmark_rate --args="addr0=192.168.10.2,addr1=192.168.110.2" --tx_rate 5e6 --tx_channels="0,2" --ref=external --pps=external
benchmark_rate --args="addr0=192.168.10.2,addr1=192.168.110.2" --tx_rate 5e6 --tx_channels="0,2"

Works as expected when DmaFIFO is disabled:
benchmark_rate --args="addr0=192.168.10.2,addr1=192.168.110.2,skip_dram=1" --tx_rate 5e6 --tx_channels="0,2"

Additional Information

See http:https://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2020-August/062553.html for background information, where the issue was narrowed down and the testing made in relation to it. Thanks to

Attached is a trace log from running benchmark_rate --args="addr0=192.168.10.2,addr1=192.168.110.2" --tx_rate 5e6 --tx_channels="0,2", where A and B outputs on the second X310 are the only ones that get enabled, instead of only both A inputs on both X310s.
uhd.log

[michaelld]

@ab-kbg Assuming this testing was on UHD 3.15.0.0 or 3.15.LTS ... yes? Please clarify. do you know if this is an issue with the recently tagged UHD4.0rc1? It's worth testing with that and reporting back here that info.

[michaelld]

Tagging @michael-west ... since some Ettus body needs to know about this ... and can hopefully comment on / verify the issue.

[ab-kbg]

@ab-kbg Assuming this testing was on UHD 3.15.0.0 or 3.15.LTS ... yes? Please clarify. do you know if this is an issue with the recently tagged UHD4.0rc1? It's worth testing with that and reporting back here that info.

Here is the full version string from UHD: GNU C++ version 10.0.1 20200328 (Red Hat 10.0.1-0.11); Boost_106900; UHD_3.15.0.0
So UHD 3.15.00, running on Fedora 32.

[ab-kbg]

@michaelld I did test 4.0.0.0-rc1 (I had to update FPGA version also) just now. linux; GNU C++ version 10.2.1 20200723 (Red Hat 10.2.1-1); Boost_106900; UHD_4.0.0.HEAD-0-g4c8ba577
The behavior of 4.0.0.0-rc1 is as expected, both with and without external sync/ref, which is great.

[michaelld]

@michaelld I did test 4.0.0.0-rc1 (I had to update FPGA version also) just now. linux; GNU C++ version 10.2.1 20200723 (Red Hat 10.2.1-1); Boost_106900; UHD_4.0.0.HEAD-0-g4c8ba577
The behavior of 4.0.0.0-rc1 is as expected, both with and without external sync/ref, which is great.

Excellent! Thanks for the feedback @ab-kbg !

[armin485]

Is there any workaround for this issue?
I'm trying transmit signals with two X310 USRPs with multi_usrp class in gnuradio but I encounter this issue.

[mbr0wn]

@armin485 workarounds include:

• Using UHD 4
• Using skip_dram=1 (if that's OK for you)
• Maybe setting the subdev spec to A:0 and then using channels=0,1 will help.

I say "maybe" because I can't reproduce this. However, the log posted has an interesting line:

2020-Aug-25 17:53:49.709397,0x7fd8402a57c0,legacy_compat.cpp:325,0,RFNOC,[legacy_compat] tx stream args: spp=364,block_id0=1/DmaFIFO_0,block_port0=1,radio_id0=0/Radio_0,radio_port0=0,block_id2=1/DmaFIFO_0,block_port2=0,radio_id2=1/Radio_0,radio_port2=0

radio0 and radio1 are correct, but the block_id0 and block_id1 are not.

[mbr0wn]

Looks like this is UHD 3 only, and has workarounds, so we're not going to handle this.