外网直连家里软路由,下载速度只有 4Mbps ~ 8Mbps 左右,使用 smb 复制文件速度一致。发现 iperf3 单线程测速的确就是这样的结果:
~ iperf3 -c xxx -R -P 1
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 7.92 MBytes 66.4 Mbits/sec
[ 5] 1.00-2.00 sec 759 KBytes 6.22 Mbits/sec
[ 5] 2.00-3.00 sec 979 KBytes 8.02 Mbits/sec
[ 5] 3.00-4.00 sec 864 KBytes 7.06 Mbits/sec
[ 5] 4.00-5.00 sec 578 KBytes 4.73 Mbits/sec
[ 5] 5.00-6.00 sec 550 KBytes 4.51 Mbits/sec
[ 5] 6.00-7.00 sec 572 KBytes 4.68 Mbits/sec
[ 5] 7.00-8.00 sec 649 KBytes 5.32 Mbits/sec
[ 5] 8.00-9.00 sec 578 KBytes 4.73 Mbits/sec
[ 5] 9.00-10.00 sec 632 KBytes 5.18 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 17.6 MBytes 14.8 Mbits/sec 1612 sender
[ 5] 0.00-10.00 sec 13.9 MBytes 11.7 Mbits/sec receiver
但神奇的是,只要改成多线程,除了宽带能跑满之外,每条线程的表现都要比单线程要好,比如 2 线程情况下,不仅能跑满 30M 的宽带,而且每条线程能到 15M 左右,比之前单线程表现要好得多:
~ iperf3 -c xxx -R -P 2
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 4.48 MBytes 37.6 Mbits/sec
[ 7] 0.00-1.00 sec 4.07 MBytes 34.2 Mbits/sec
[SUM] 0.00-1.00 sec 8.55 MBytes 71.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 2.02 MBytes 17.0 Mbits/sec
[ 7] 1.00-2.00 sec 1.50 MBytes 12.6 Mbits/sec
[SUM] 1.00-2.00 sec 3.53 MBytes 29.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 2.02 MBytes 17.0 Mbits/sec
[ 7] 2.00-3.00 sec 1.49 MBytes 12.5 Mbits/sec
[SUM] 2.00-3.00 sec 3.51 MBytes 29.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 2.14 MBytes 17.9 Mbits/sec
[ 7] 3.00-4.00 sec 1.33 MBytes 11.2 Mbits/sec
[SUM] 3.00-4.00 sec 3.47 MBytes 29.1 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 1.90 MBytes 16.0 Mbits/sec
[ 7] 4.00-5.00 sec 1.57 MBytes 13.1 Mbits/sec
[SUM] 4.00-5.00 sec 3.47 MBytes 29.1 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 2.05 MBytes 17.2 Mbits/sec
[ 7] 5.00-6.00 sec 1.48 MBytes 12.4 Mbits/sec
[SUM] 5.00-6.00 sec 3.53 MBytes 29.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 1.92 MBytes 16.1 Mbits/sec
[ 7] 6.00-7.00 sec 1.54 MBytes 12.9 Mbits/sec
[SUM] 6.00-7.00 sec 3.46 MBytes 29.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 1.98 MBytes 16.6 Mbits/sec
[ 7] 7.00-8.00 sec 1.56 MBytes 13.1 Mbits/sec
[SUM] 7.00-8.00 sec 3.55 MBytes 29.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 1.88 MBytes 15.8 Mbits/sec
[ 7] 8.00-9.00 sec 1.51 MBytes 12.6 Mbits/sec
[SUM] 8.00-9.00 sec 3.39 MBytes 28.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 1.97 MBytes 16.5 Mbits/sec
[ 7] 9.00-10.00 sec 1.59 MBytes 13.3 Mbits/sec
[SUM] 9.00-10.00 sec 3.56 MBytes 29.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 26.0 MBytes 21.8 Mbits/sec 9579 sender
[ 5] 0.00-10.00 sec 22.4 MBytes 18.8 Mbits/sec receiver
[ 7] 0.00-10.01 sec 21.1 MBytes 17.7 Mbits/sec 8230 sender
[ 7] 0.00-10.00 sec 17.6 MBytes 14.8 Mbits/sec receiver
[SUM] 0.00-10.01 sec 47.1 MBytes 39.5 Mbits/sec 17809 sender
[SUM] 0.00-10.00 sec 40.0 MBytes 33.6 Mbits/sec receiver
基于这个结论,尝试打开一个 SMB 文件复制的同时,开启 iperf3 单线程测试,此时 iperf3 也能跑到 15M 以上,同时 SMB 复制速度也提上去了
~ iperf3 -c xxx -R -P 1
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 6.47 MBytes 54.3 Mbits/sec
[ 5] 1.00-2.00 sec 1.87 MBytes 15.7 Mbits/sec
[ 5] 2.00-3.00 sec 1.73 MBytes 14.5 Mbits/sec
[ 5] 3.00-4.00 sec 1.69 MBytes 14.2 Mbits/sec
[ 5] 4.00-5.00 sec 1.82 MBytes 15.2 Mbits/sec
[ 5] 5.00-6.00 sec 1.97 MBytes 16.5 Mbits/sec
[ 5] 6.00-7.00 sec 1.92 MBytes 16.2 Mbits/sec
[ 5] 7.00-8.00 sec 1.67 MBytes 14.0 Mbits/sec
[ 5] 8.00-9.00 sec 1.56 MBytes 13.1 Mbits/sec
[ 5] 9.00-10.00 sec 1.58 MBytes 13.3 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 25.8 MBytes 21.6 Mbits/sec 8644 sender
[ 5] 0.00-10.00 sec 22.3 MBytes 18.7 Mbits/sec receiver
所以有个骚操作,如果想提升 SMB 的复制速度,可以同时开一个 5 线程 1M 的 iperf3 测速,这样 SMB + iperf3 能把带宽打满,同时 iperf3 只占用 5M 左右带宽
~ iperf3 -c xxx -R -P 5 -b 1M
最后问题来了,为什么会出现这样的情况?软路由要调整什么配置才能恢复单线程性能? 另外补充一个信息,同网络下另外一台 PC 主机并没有出现这种情况,PC 的单线程也能把宽带跑满
1
serafin 2023-06-23 04:51:52 +08:00
修改 MTU 看看
|
2
txydhr 2023-06-23 08:22:26 +08:00
什么 cpu ?
|
3
ayelky 2023-06-23 09:50:29 +08:00
ping?
|
4
ryd994 2023-06-23 10:04:55 +08:00 via Android
很正常,软路由包处理很吃 CPU 性能。单个连接只会由同一个 CPU 处理(否则顺序会乱)。多连接,只要不是运气很差,可以由不同的 CPU 处理,这就是两倍性能了。
|
5
ryd994 2023-06-23 10:06:20 +08:00 via Android
mmm ,你说的这个现象可能和睿频有关
单核负荷不够高,系统以为不需要提高频率 |
6
HaEx OP 感谢老哥们提供的思路,我把核心和频率都打开再试试
|
7
HaEx OP @txydhr S905x3 ,本体是个电视盒子,刷了 CoreELEC ,除了软路由,上面也同时跑了不少应用
|
8
HaEx OP @ryd994 把核心和频率都打开后,现象一致并没有改变,所以跟 CPU 无关,实际上 iperf3 在跑的时候,CPU 负载非常低
|
9
HaEx OP @serafin 测试了下,iperf3 的 mss 设置调低之后,对单线程和多线程都有明显影响,但手动设置最高只有 1024 ,此时表现跟不设置是一样的,所以不确定 mtu 是否是最终影响因素。另外网卡自身的 mtu 是 1500
|
12
ryd994 2023-06-24 01:26:01 +08:00 via Android
@HaEx 你是说用小 mss 反而可以提高带宽?
那可能和网卡的 interrupt moderation 有关,如果不能关闭网卡的节能和 interrupt moderation 的话,可以尝试启用 busy_poll 和换用不同的 congestion control |
13
fengyaochen 2023-06-24 10:30:37 +08:00 via iPhone
开 bbr 节能 mtu mss 感觉影响不大,除非你是 vpn 速度不行
|
14
LGA1150 2023-06-24 22:14:33 +08:00 via Android
iperf3 UDP 测速(-b 参数指定发送速度),能跑到多少不丢包?
|
15
HaEx OP @fengyaochen ipv6 直连,没有 VPN
|
18
HaEx OP @LGA1150 udp 三线程 30M 基本不丢包,单线程最高只能跑到 16M 左右
``` ~ iperf3 -c xxx -R -u -b 10m -P 3 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 12.1 MBytes 10.1 Mbits/sec 0.689 ms 0/1543 (0%) [ 4] Sent 1543 datagrams [ 6] 0.00-10.00 sec 12.1 MBytes 10.1 Mbits/sec 0.441 ms 0/1543 (0%) [ 6] Sent 1543 datagrams [ 8] 0.00-10.00 sec 12.1 MBytes 10.1 Mbits/sec 0.672 ms 0/1543 (0%) [ 8] Sent 1543 datagrams [SUM] 0.00-10.00 sec 36.2 MBytes 30.3 Mbits/sec 0.601 ms 0/4629 (0%) ``` ``` ~ iperf3 -c xxx -R -u -b 100m -P 1 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 20.0 MBytes 16.8 Mbits/sec 0.481 ms 0/2558 (0%) [ 4] Sent 2558 datagrams ``` |
19
HaEx OP 应该是 tcp 拥塞算法问题,从 bbr 改到 cubic 之后问题消失了,再观察一段时间看看
|