trafgen: not sending all the packets specified by num

[jonschipp]

trafgen -v

trafgen 0.5.8-rc2+ (Ziggomatic), Git id: v0.5.8-rc2-26-gc2887b3

I'm having two separate issues.
I'm having an issue with trafgen where the only way I can get it to generate the exact number of packets specified by --num is if I do this AND by issuing it to the shell in contrast from using the same command multiple times in script:

$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num 15000000
--kernel-pull 6 works as well.

I've tried various kernel pulls, CPU, and ring size combinations and found that the
above is the only one where my system will generate all 15000000 packets.

I'm reporting no tx loss on the sending host. I've been watching /proc/net/dev and ethtool and do not see any tx discards or drops.

If I launch trafgen with the same arguments and options from a shell script or atd job
the number of packets actually sent vary greatly. For my tests I'm running trafgen
every 3 minutes. The script is executed via crond on the sending host every hour.

!/bin/bash

sleep 10

for i in {0..29..3}
do

echo "trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num 15000000" | at now +${i}min

done

The receiving and sending machine's /proc/net/dev packet column
show the following values. 1 for each of 10 runs per tool is listed.
I'm not sure why the first trafgen instance for each tool sends the full
15000000 packets and the rest slack off.

root@recv:~/64# grep ^eth2: *.stat | awk '{ print $1, $3 }'
daemonlogger.stat:eth2: 15000000
daemonlogger.stat:eth2: 13691387
daemonlogger.stat:eth2: 13441450
daemonlogger.stat:eth2: 13884259
daemonlogger.stat:eth2: 13757665
daemonlogger.stat:eth2: 13714474
daemonlogger.stat:eth2: 13581056
daemonlogger.stat:eth2: 13530621
daemonlogger.stat:eth2: 13562485
daemonlogger.stat:eth2: 13460224
dumpcap.stat:eth2: 15000000
dumpcap.stat:eth2: 13984965
dumpcap.stat:eth2: 13974693
dumpcap.stat:eth2: 13772195
dumpcap.stat:eth2: 13914422
dumpcap.stat:eth2: 14044146
dumpcap.stat:eth2: 13782769
dumpcap.stat:eth2: 13813973
dumpcap.stat:eth2: 13869773
dumpcap.stat:eth2: 13756627
...

Here's two tests to demonstrate the difference between the # packets sent on my machine and the the trafgen options used. The /proc/net/dev output below is on the sending machine from which trafgen is running.

$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num 15000000

eth2: 0 0 0 0 0 0 0 0 1020000000 15000000 0 0 0 0 0 0

$ trafgen --in configs/64.cfg --out eth2 --verbose --num 15000000

eth2: 0 0 0 0 0 0 0 0 1016840720 14953540 0 0 0 0 0 0

Here's a diff between the ethtool output of each:

$ diff good.ethtool.txt bad.ethtool.txt
28c28

< tx_octets: 1020000000

 tx_octets: 1016840720

53c53

< tx_ucast_packets: 15000000

 tx_ucast_packets: 14953540

65,66c65,66
< dma_readq_full: 5006163

< dma_read_prioq_full: 0

 dma_readq_full: 104878
 dma_read_prioq_full: 235

68,72c68,72
< ring_set_send_prod_index: 15000000
< ring_status_update: 1143980
< nic_irqs: 488244
< nic_avoided_irqs: 655736

< nic_tx_threshold_hit: 1118132

 ring_set_send_prod_index: 14952642
 ring_status_update: 2150811
 nic_irqs: 367275
 nic_avoided_irqs: 1783536
 nic_tx_threshold_hit: 2124554

[borkmann]

Thanks for reporting Jon!

How does configs/64.cfg look ? Could you copy its content here ?

Is the same issue when you specify -t0 ?

What if you leave out --cpus option ?

[jonschipp]

The config I'm using is generated by gencfg with the -G rfc2544 option.
The only difference is the MAC and IP addresses from below are replaced with the ones of my two servers.
Just an FYI: I am using UDP Src and Dst port 0 and for no particular reason other than I figured no ICMP
port unreachable would come back from the receiver since I don't think you can even create a socket bound
to port 0.

$ cat 64.cfg
/* RFC2544 - Frame Size: 64 /
{
/ Dst Mac /
0xff,0xff,0xff,0xff,0xff,0xff,
/ Src Mac /
0x11,0x22,0x33,0x440,0x55,0x66,
/ EtherType /
c16(0x0800),
/ IPv4 Version, IHL, TOS /
0b01000101, 0,
/ IPv4 Total Len /
c16(50),
/ IPv4 Ident /
drnd(2),
/ IPv4 Flags, Frag Off /
0b01000000, 0,
/ IPv4 TTL /
64,
/ Proto UDP /
17,
/ IPv4 Checksum (IP header from, to) /
csumip(14, 33),
/ Source IP /
192,168,1,255,
/ Dest IP /
192,168,1,1,
/ UDP Source Port /
c16(0),
/ UDP Dest Port /
c16(0),
/ Length /
c16(30),
/ Checksum /
c16(00),
/ Data */
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
0xff,
}

I have tried leaving out --cpus while keeping --kernel-pull 5 but I don't remember what my results were. I think the same but I will test again tonight. I have not tried messing with the gap, is the default not 0? I'll try tonight.

[jonschipp]

My results were better today.

* indicate the number of runs I did for each test. # PPS is the rough packet rate.

eth5 - e1000

$ trafgen --in configs/64.cfg --out eth5 --verbose --kernel-pull 5 --cpus 1 --num 15000000

174401 PPS

3 * 15000000 (Send and Recv)

$ trafgen --in configs/64.cfg --out eth2 --verbose --num 15000000

229,000 PPS

2 * 15000000 (Send and Recv)
1 * 14999106 (Send and Recv)
1 * 14997431 (Send and Recv)

$ trafgen --in configs/64.cfg --out eth5 --verbose --kernel-pull 5 --num 15000000

10 PPS

rx: 0.000 MiB/t 0 pkts/t 0 drops/t 0 errors/t
tx: 0.001 MiB/t 11 pkts/t 0 drops/t 0 errors/t

trafgen --in configs/64.cfg --out eth5 --verbose --kernel-pull 5 --num 15000000
[0] pkt
len 64 cnts 0 rnds 2
payload 00 14 38 4c 91 46 00 04 23 c8 77 cc 08 00 45 00 00 32 00 00 40 00 40 11
00 00 0a 7d f3 a4 0a 03 8a d3 00 00 00 00 00 1e 00 00 ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff
rnd0 off 18
rnd1 off 19
4 packets to schedule
256 bytes in total
Running! Hang up with ^C!

TX,V2: 238.41 MiB, 122064 Frames, each 2048 Byte allocated
TX,V2: 238.41 MiB, 122064 Frames, each 2048 Byte allocated
TX,V2: 238.41 MiB, 122064 Frames, each 2048 Byte allocated
TX,V2: 238.41 MiB, 122064 Frames, each 2048 Byte allocated
^C
862 packets outgoing
55168 bytes outgoing
50 sec, 754602 usec on CPU0 (3 packets)
50 sec, 185178 usec on CPU1 (662 packets)
50 sec, 100031 usec on CPU2 (193 packets)
24 sec, 71074 usec on CPU3 (4 packets)

$ trafgen --in configs/65.cfg --out eth5 --verbose --kernel-pull 5 --cpus 1 -t 0 --num 15000000

3 * 15000000 (Send and Recv)

468689 PPS

$ trafgen --in configs/64.cfg --out eth5 --verbose -t 0 --num 15000000

3 * 15000000 (Send and Recv)

177012 PPS

$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 -t 0 --num 15000000

3 * 15000000 (Send and Recv)

176821 PPS

=== eth2 - tg3

$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num

3 * 15000000 (Send and Recv)

196817 PPS

$ trafgen --in configs/64.cfg --out eth2 --verbose --num 15000000

1 * 14978089 (Send and Recv)
1 * 14998516 (Send and Recv)
1 * 14994182 (Send and Recv)

223788 PPS

$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --num 15000000

rx: 0.000 MiB/t 0 pkts/t 0 drops/t 0 errors/t
tx: 0.001 MiB/t 10 pkts/t 0 drops/t 0 errors/t

$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 -t 0 --num 15000000

3 * 15000000 (Send and Recv)

335313 PPS

$ trafgen --in configs/64.cfg --out eth2 --verbose -t 0 --num 15000000

3 * 15000000 (Send and Recv)

151974 PPS

$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 -t 0 --num 15000000

3 * 15000000

150403 PPS

[jonschipp]

On two beefier machines, running base install of Debian, trafgen still has the same issue when specifying --num if it's called from a shell script or at job doesn't generate all the packets. Though, on the command-line, for these systems, I don't need to use the kernel pull option and I get my best PPS rate when using -t 0 and binding to a single cpu just like on the older machines.

I'll test the examples from my last post on the system soon.

[jonschipp]

Tests on new system, Dell Poweredge 1950.

$ uname -a
Linux send-pe 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1+deb7u1 x86_64 GNU/Linux

[ 3100.362602] bnx2 0000:07:00.0: eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem f4000000, IRQ 16, node addr 00:1e:c9:b2:d5:41
[ 3100.374080] bnx2 0000:07:00.0: firmware: agent loaded bnx2/bnx2-mips-06-6.2.1.fw into memory
[ 3100.377009] bnx2 0000:07:00.0: firmware: agent loaded bnx2/bnx2-rv2p-06-6.0.15.fw into memory
[ 3100.378851] bnx2 0000:07:00.0: irq 48 for MSI/MSI-X
[ 3100.492011] bnx2 0000:07:00.0: eth1: using MSI

$ ethtool -i eth1
driver: bnx2
version: 2.1.11
firmware-version: bc 4.0.3 ipms 1.6.0
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 23
Stepping: 6
CPU MHz: 2327.919
BogoMIPS: 4655.05
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 6144K
NUMA node0 CPU(s): 0-7

$ trafgen --in configs/64.cfg --out eth1 --verbose --num 15000000
1 * 14147986
1 * 14269021
1 * 14147333

$ trafgen --in configs/64.cfg --out eth1 --verbose --kernel-pull 5 --num 15000000
1 * 14965689
1 * 14955465
1 * 14930939

$ trafgen --in configs/64.cfg --out eth1 --verbose --kernel-pull 5 -t 0 --num 15000000
3 * 15000000

$ trafgen --in configs/64.cfg --out eth1 --verbose -t 0 --num 15000000
3 * 15000000

[jonschipp]

FYI: I installed the latest mainline/vanilla kernel, 3.11.0, and noticed that trafgen sends more packets than requested by --num. This is the first time I've seen this. With this came a new module, the driver version is now 2.2.3 instead of 2.1.11. I don't know if this could be related or not.

$ ethtool -i eth1
driver: bnx2
version: 2.2.3
firmware-version: bc 4.0.3 ipms 1.6.0
...

$ trafgen --in configs/64.cfg --out eth1 --verbose --cpus 1 -t 0 --num 15000000

1 * 15000009
1 * 15000015
1 * 15000004

[jonschipp]

FYI: I'm using pktgen in the same way without problems so I don't think it can be a network driver issue
i.e. it generates 15 million 64 byte frames, no more and no less. Works from a script too.