Unlimited thoughput during tcset

[twdkeule]

HI,
I am using tcset to limit the throughput of a connection while following a bandwidth profile. For this I use the tcset command every 30 seconds. However, I notice a spike in the available throughput right after/while the new limit is set. I assume that the previous limit is removed, before adding a new limit and between these there is a not insignificant amount of time when there is no limit.

Is there a possibility to avoid this period of unlimited bandwidth?

As you can see, the measured throughput (red) spikes after when the available throughput (green) has changed (@ 60, 150 and 210 seconds)

[thombashi]

Thank you for sharing interesting information.

I assume that the previous limit is removed, before adding a new limit and between these there is a not insignificant amount of time when there is no limit.

If your assumption is correct, the following might workaround the issue
(these will changing the existing tc bandwidth setting instead of delete and add).

Would you try this?

1. Initial setup

# tcset --device eth0 --rate 6M 

2. Create a tc script for the next bandwidth

# tcset --device eth0 --rate 5M --tc-script
[INFO] tcconfig: written a tc script to 'tcset_eth0.sh'

# cat tcset_eth0.sh
#!/bin/sh

# tc script file:
#   the following command sequence lead to equivalent results as
#   'tcset --device eth0 --rate 5M'.
#   created by tcset on 2017-06-07T00:09:14.

tc qdisc add dev eth0 root handle 1a1a: htb default 1
tc class add dev eth0 parent 1a1a: classid 1a1a:1 htb rate 1000000kbit
tc class add dev eth0 parent 1a1a: classid 1a1a:2 htb rate 5000.0Kbit ceil 5000.0Kbit burst 62.5KB cburst 62.5KB
tc qdisc add dev eth0 parent 1a1a:2 handle 1a9a: netem
tc filter add dev eth0 protocol ip parent 1a1a: prio 1 u32 match ip dst 0.0.0.0/0 flowid 1a1a:2

3. Modify the script

• Comment out the first and the last line
• Change add subcommand to change

# cat tcset_eth0.sh
#!/bin/sh

#tc qdisc add dev eth0 root handle 1a1a: htb default 1
tc class change dev eth0 parent 1a1a: classid 1a1a:1 htb rate 1000000kbit
tc class change dev eth0 parent 1a1a: classid 1a1a:2 htb rate 5000.0Kbit ceil 5000.0Kbit burst 62.5KB cburst 62.5KB
tc qdisc change dev eth0 parent 1a1a:2 handle 1a9a: netem
#tc filter add dev eth0 protocol ip parent 1a1a: prio 1 u32 match ip dst 0.0.0.0/0 flowid 1a1a:2

4. Execute the modified script

# ./tcset_eth0.sh

[twdkeule]

I forgot to mention but it might be important due to the high throughput or low latency, but I was running this over the loopback interface

[twdkeule]

This looks better!

The first iperf3 session is while executing sudo tcset --device lo --rate 4M --overwrite. Notice the spike to 2.2Gbps.
The second session while executing the adapted script. There is no spike!

➜  ~ iperf3 -c 127.0.0.1 -O 1
Connecting to host 127.0.0.1, port 5201
[  4] local 127.0.0.1 port 43952 connected to 127.0.0.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.12 MBytes  26.2 Mbits/sec    0    639 KBytes       (omitted)
[  4]   0.00-1.00   sec   767 KBytes  6.28 Mbits/sec    0    639 KBytes       
[  4]   1.00-2.00   sec   767 KBytes  6.29 Mbits/sec    0    639 KBytes       
[  4]   2.00-3.00   sec   639 KBytes  5.24 Mbits/sec    0    639 KBytes       
[  4]   3.00-4.00   sec  20.0 MBytes   168 Mbits/sec    2    639 KBytes       
[  4]   4.00-5.00   sec   268 MBytes  2.25 Gbits/sec    1   1.12 MBytes       
[  4]   5.00-6.00   sec   639 KBytes  5.24 Mbits/sec    0   1.12 MBytes       
[  4]   6.00-7.00   sec   448 KBytes  3.67 Mbits/sec    0   1.12 MBytes       
[  4]   7.00-8.00   sec   448 KBytes  3.67 Mbits/sec    0   1.12 MBytes       
[  4]   8.00-9.00   sec   512 KBytes  4.19 Mbits/sec    0   1.12 MBytes       
[  4]   9.00-10.00  sec   448 KBytes  3.67 Mbits/sec    0   1.12 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   293 MBytes   245 Mbits/sec    3             sender
[  4]   0.00-10.00  sec   291 MBytes   244 Mbits/sec                  receiver

iperf Done.
➜  ~ iperf3 -c 127.0.0.1 -O 1
Connecting to host 127.0.0.1, port 5201
[  4] local 127.0.0.1 port 44024 connected to 127.0.0.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  2.94 MBytes  24.6 Mbits/sec    0    639 KBytes       (omitted)
[  4]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec    0    639 KBytes       
[  4]   1.00-2.00   sec   448 KBytes  3.67 Mbits/sec    0    639 KBytes       
[  4]   2.00-3.00   sec   512 KBytes  4.19 Mbits/sec    0    639 KBytes       
[  4]   3.00-4.00   sec   512 KBytes  4.20 Mbits/sec    0    639 KBytes       
[  4]   4.00-5.00   sec   639 KBytes  5.24 Mbits/sec    0    639 KBytes       
[  4]   5.00-6.00   sec   576 KBytes  4.71 Mbits/sec    0    639 KBytes       
[  4]   6.00-7.00   sec   639 KBytes  5.24 Mbits/sec    0    639 KBytes       
[  4]   7.00-8.00   sec   576 KBytes  4.72 Mbits/sec    0    639 KBytes       
[  4]   8.00-9.00   sec   639 KBytes  5.24 Mbits/sec    0    639 KBytes       
[  4]   9.00-10.00  sec   576 KBytes  4.72 Mbits/sec    0    639 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  5.50 MBytes  4.61 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  5.68 MBytes  4.77 Mbits/sec                  receiver

iperf Done.
➜  ~ 

[thombashi]

Thank you for sharing the results.

Then, I will modify the tcset --overwrite behavior to use change tc settings instead of delete and add.
Please wait for a moment till next release.

[twdkeule]

Thanks!
I wrote a python script to change the 'add' to 'change' and remove the filter and root lines, so I'm good for now.

[thombashi]

I've added --change option at tcconfig 0.12.0:

# tcset --device eth0 --rate 6M
# tcset --device eth0 --rate 5M --change

This option will use change internally instead of delete and add.

[twdkeule]

Super!

[thombashi]

I'll close the issue then.
Thanks,