東京リージョンと大阪リージョンのEC2インスタンス間の転送速度を計測してみた
マルチフロートラフィックならEC2インスタンス間で70Gbpsは出せることが分かりました
2024.05.15 01:26東京リージョンと大阪リージョンのEC2インスタンス間でどのぐらいの速度で転送できるのか気になる
こんにちは、のんピ(@non____97)です。
皆さんは東京リージョンと大阪リージョンのEC2インスタンス間でどのぐらいの転送速度が出るのか気になったことはありますか? 私はあります。
リージョン間の通信であってもvCPUが32個以上あるEC2インスタンスであればインスタンスで利用可能なネットワーク帯域幅の最大50％を利用できるため、それなりの転送速度が出そうです。
マルチフロートラフィック
インスタンスで使用できる集約マルチフロートラフィックの帯域幅は、トラフィックの宛先によって異なります。
リージョン内 – トラフィックは、インスタンスで使用可能な全ネットワーク帯域幅を利用することができます。
他のリージョンへの通信、インターネットゲートウェイ、Direct Connect、ローカルゲートウェイ (LGW) – トラフィックは、最低 32 個の vCPUs を搭載した現行世代のインスタンスで利用可能なネットワーク帯域幅の最大 50％ を利用できます。32vCPUs未満の現世代のインスタンスの帯域幅は 5 Gbps に制限されています。
シングルフロートラフィック
インスタンスが同じクラスタープレイスメントグループにない場合、シングルフロートラフィックのベースライン帯域幅は 5 Gbps に制限されます。レイテンシーを減らし、シングルフロー帯域幅を増やすには、以下のいずれかをお試しください。
クラスタープレイスメントグループを使用すると、同じプレイスメントグループ内のインスタンスで最大 10 Gbps の帯域幅を実現できます。
または、任意の 2 つのエンドポイント間に複数のパスを設定することで、Multipath TCP (MPTCP) を使用して高帯域幅を実現できます。
同じサブネット内の対象インスタンスに ENA Express を設定して、それらのインスタンス間で最大 25 Gbps を実現します。
実際にどのぐらいの転送速度が出るのか試したことがなかったので、試してみました。
いきなりまとめ
- リージョンを跨いだ単一インスタンス間でも、マルチフロートラフィックであれば高転送レートで転送できる
- Transit Gateway経由の場合はバーストで50Gbpsまでしか出ないので注意
- c6gn.mediumの場合 (単位はGbps)
- c6gn.mediumのネットワーク帯域幅は最大16Gbps
| プロトコル (バッファサイズ調整有無) / 多重度 | 1 | 4 | 8 |
|---|---|---|---|
| TCP (バッファサイズ調整なし) | 2.92 | - | - |
| TCP (バッファサイズ調整あり) | 4.75 | 11.7 | 11.7 |
| UDP (バッファサイズ調整なし) | 2.19 | - | - |
| UDP (バッファサイズ調整あり) | 2.56 | 3.39 | 3.59 |
- c6gn.12xlargeの場合 (単位はGbps)
- c6gn.12xlargeのネットワーク帯域幅は75Gbps
| プロトコル (バッファサイズ調整有無) / 多重度 | 1 | 4 | 8 | 16 | 32 | 64 | 96 |
|---|---|---|---|---|---|---|---|
| TCP (バッファサイズ調整なし) | 2.95 | 12.2 | 24.5 | 47.4 | 66.4 | 71 | 71.2 |
| TCP (バッファサイズ調整あり) | 4.75 | - | - | 56.7 | 69.8 | 71.1 | - |
| UDP (バッファサイズ調整なし) | 4.75 | 19.1 | - | - | - | - | - |
| UDP (バッファサイズ調整あり) | - | - | - | 38.5 | - | 53 | - |
検証環境
検証環境は以下のとおりです。
- OS : Amazon Linux 2023 (al2023-ami-2023.4.20240429.0-kernel-6.1-arm64)
- インスタンスタイプ : c6gn.medium (ネットワーク帯域幅最大16Gbps)
東京リージョンのEC2インスタンスから大阪リージョンのEC2インスタンスへiperf3を叩いて、その時の転送速度を確認します。iperf3の詳細は以下記事をご覧ください。
レイテンシーは8msで安定しています。
EC2インスタンスの各種設定は以下のとおりです。
$ sudo cpupower frequency-info
analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
$ sudo sysctl -a | grep -e wmem -e rmem -e mtu -e default_qdis
net.core.default_qdisc = fq_codel
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.ipv4.ip_forward_use_pmtu = 0
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.tcp_mtu_probe_floor = 48
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_rmem = 4096 131072 6291456
net.ipv4.tcp_wmem = 4096 20480 4194304
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv6.conf.all.accept_ra_mtu = 1
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.default.accept_ra_mtu = 1
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.ens5.accept_ra_mtu = 1
net.ipv6.conf.ens5.mtu = 9001
net.ipv6.conf.lo.accept_ra_mtu = 1
net.ipv6.conf.lo.mtu = 65536
net.ipv6.route.mtu_expires = 600
vm.lowmem_reserve_ratio = 256 256 32 0 0
$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 0a:49:24:4f:07:c5 brd ff:ff:ff:ff:ff:ff
altname enp0s5
altname eni-07615f08d54aa652f
altname device-number-0
$ ethtool -g ens5
Ring parameters for ens5:
Pre-set maximums:
RX: 2048
RX Mini: n/a
RX Jumbo: n/a
TX: 512
Current hardware settings:
RX: 1024
RX Mini: n/a
RX Jumbo: n/a
TX: 512
$ cat /sys/devices/system/cpu/smt/active
0
c6gn.medium の場合
TCP 多重度 1
まずはTCPで試します。
大阪リージョンのEC2インスタンスで待ち受けておきます。
$ iperf3 -s -V iperf 3.16 Linux ip-10-0-0-7.ap-northeast-3.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64 ----------------------------------------------------------- Server listening on 5201 (test #1) -----------------------------------------------------------
東京側から接続します。
$ iperf3 -c 10.0.0.7 -V
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 07:57:58 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: jvx6y2phl7vnzjwr3u3zfozqwystnleoddkf
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 56460 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 320 MBytes 2.68 Gbits/sec 0 6.03 MBytes
[ 5] 1.00-2.00 sec 351 MBytes 2.94 Gbits/sec 0 6.03 MBytes
[ 5] 2.00-3.00 sec 351 MBytes 2.94 Gbits/sec 0 6.03 MBytes
[ 5] 3.00-4.00 sec 348 MBytes 2.92 Gbits/sec 0 6.03 MBytes
[ 5] 4.00-5.00 sec 351 MBytes 2.94 Gbits/sec 0 6.03 MBytes
[ 5] 5.00-6.00 sec 352 MBytes 2.95 Gbits/sec 0 6.03 MBytes
[ 5] 6.00-7.00 sec 352 MBytes 2.96 Gbits/sec 0 6.03 MBytes
[ 5] 7.00-8.00 sec 351 MBytes 2.95 Gbits/sec 0 6.03 MBytes
[ 5] 8.00-9.00 sec 353 MBytes 2.96 Gbits/sec 0 6.03 MBytes
[ 5] 9.00-10.00 sec 352 MBytes 2.95 Gbits/sec 0 6.03 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.40 GBytes 2.92 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 3.40 GBytes 2.92 Gbits/sec receiver
CPU Utilization: local/sender 10.6% (0.2%u/10.4%s), remote/receiver 20.3% (0.4%u/19.9%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
2.92GBでした。
大阪側でも同様の値を示しています。
$ iperf3 -s -V
iperf 3.16
Linux ip-10-0-0-7.ap-northeast-3.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Time: Tue, 14 May 2024 07:57:58 GMT
Accepted connection from 172.31.8.203, port 56450
Cookie: jvx6y2phl7vnzjwr3u3zfozqwystnleoddkf
TCP MSS: 0 (default)
[ 5] local 10.0.0.7 port 5201 connected to 172.31.8.203 port 56460
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 317 MBytes 2.65 Gbits/sec
[ 5] 1.00-2.00 sec 350 MBytes 2.94 Gbits/sec
[ 5] 2.00-3.00 sec 352 MBytes 2.95 Gbits/sec
[ 5] 3.00-4.00 sec 348 MBytes 2.92 Gbits/sec
[ 5] 4.00-5.00 sec 350 MBytes 2.94 Gbits/sec
[ 5] 5.00-6.00 sec 352 MBytes 2.95 Gbits/sec
[ 5] 6.00-7.00 sec 352 MBytes 2.96 Gbits/sec
[ 5] 7.00-8.00 sec 352 MBytes 2.95 Gbits/sec
[ 5] 8.00-9.00 sec 352 MBytes 2.96 Gbits/sec
[ 5] 9.00-10.00 sec 352 MBytes 2.96 Gbits/sec
[ 5] 10.00-10.01 sec 2.88 MBytes 2.85 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate
[ 5] (sender statistics not available)
[ 5] 0.00-10.01 sec 3.40 GBytes 2.92 Gbits/sec receiver
rcv_tcp_congestion cubic
それなりに速いと言えるのではないでしょうか。
何回か繰り返しましたが、いずれも3Gbpsほどでした。
UDP 多重度 1
続いてUDPでも試します。
$ iperf3 -c 10.0.0.7 -V -u -b 16G
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:08:20 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: htq536yhyuxz7xl2ideoqrfkn3sz2pxvckfb
Target Bitrate: 16000000000
[ 5] local 172.31.8.203 port 48620 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 580 MBytes 4.87 Gbits/sec 420065
[ 5] 1.00-2.00 sec 579 MBytes 4.86 Gbits/sec 419572
[ 5] 2.00-3.00 sec 579 MBytes 4.86 Gbits/sec 419517
[ 5] 3.00-4.00 sec 579 MBytes 4.86 Gbits/sec 419578
[ 5] 4.00-5.00 sec 579 MBytes 4.86 Gbits/sec 419564
[ 5] 5.00-6.00 sec 579 MBytes 4.86 Gbits/sec 419549
[ 5] 6.00-7.00 sec 579 MBytes 4.86 Gbits/sec 419534
[ 5] 7.00-8.00 sec 579 MBytes 4.86 Gbits/sec 419573
[ 5] 8.00-9.00 sec 579 MBytes 4.86 Gbits/sec 419558
[ 5] 9.00-10.00 sec 579 MBytes 4.85 Gbits/sec 419548
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 5.66 GBytes 4.86 Gbits/sec 0.000 ms 0/4196058 (0%) sender
[ 5] 0.00-10.01 sec 2.55 GBytes 2.19 Gbits/sec 0.002 ms 2301865/4196058 (55%) receiver
CPU Utilization: local/sender 74.3% (5.5%u/68.8%s), remote/receiver 94.4% (10.9%u/83.5%s)
iperf Done.
2.19Gbpsですね。ロストした量がそれなりに多いですね。
大阪側も2.19Gbpsで記録されていました。
iperf 3.16
Linux ip-10-0-0-7.ap-northeast-3.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
-----------------------------------------------------------
Server listening on 5201 (test #4)
-----------------------------------------------------------
Time: Tue, 14 May 2024 08:01:08 GMT
Accepted connection from 172.31.8.203, port 53226
Cookie: vrvvnw4nxsviukpsqqa72jbw6e44rv6odptl
Target Bitrate: 16000000000
[ 5] local 10.0.0.7 port 5201 connected to 172.31.8.203 port 50881
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 268 MBytes 2.25 Gbits/sec 0.001 ms 221974/416088 (53%)
[ 5] 1.00-2.00 sec 275 MBytes 2.30 Gbits/sec 0.001 ms 221285/420138 (53%)
[ 5] 2.00-3.00 sec 275 MBytes 2.30 Gbits/sec 0.001 ms 218408/417247 (52%)
[ 5] 3.00-4.00 sec 275 MBytes 2.31 Gbits/sec 0.001 ms 220354/419700 (53%)
[ 5] 4.00-5.00 sec 240 MBytes 2.01 Gbits/sec 0.007 ms 245757/419428 (59%)
[ 5] 5.00-6.00 sec 239 MBytes 2.00 Gbits/sec 0.002 ms 246635/419639 (59%)
[ 5] 6.00-7.00 sec 274 MBytes 2.30 Gbits/sec 0.002 ms 221069/419825 (53%)
[ 5] 7.00-8.00 sec 240 MBytes 2.01 Gbits/sec 0.001 ms 244830/418634 (58%)
[ 5] 8.00-9.00 sec 264 MBytes 2.22 Gbits/sec 0.001 ms 228876/420104 (54%)
[ 5] 9.00-10.00 sec 260 MBytes 2.18 Gbits/sec 0.001 ms 232263/420800 (55%)
[ 5] 10.00-10.01 sec 3.59 MBytes 3.40 Gbits/sec 0.008 ms 1845/4446 (41%)
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] (sender statistics not available)
[ 5] 0.00-10.01 sec 2.55 GBytes 2.19 Gbits/sec 0.008 ms 2303296/4196049 (55%) receiver
iperf 3.16
Linux ip-10-0-0-7.ap-northeast-3.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
バッファサイズの変更をしたのち、UDP 多重度 1
大阪のEC2インスタンスを確認すると、バッファが溢れていそうです。
$ netstat -su
IcmpMsg:
InType3: 3
OutType3: 3
Udp:
1894261 packets received
3 packets to unknown port received
2301503 packet receive errors
72 packets sent
2301503 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InOctets: 6193293239
OutOctets: 157453
InNoECTPkts: 4197024
MPTcpExt:
大阪のEC2インスタンスのバッファサイズを変更してあげます。
$ sudo sysctl -w net.ipv4.tcp_wmem="65536 33554432" $ sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 33554432" $ sudo sysctl -w net.core.wmem_default=67108864 $ sudo sysctl -w net.core.rmem_default=67108864 $ sudo sysctl -w net.core.wmem_max=67108864 $ sudo sysctl -w net.core.rmem_max=67108864 $ sudo sysctl -a | grep -e wmem -e rmem net.core.rmem_default = 2147483647 net.core.rmem_max = 2147483647 net.core.wmem_default = 2147483647 net.core.wmem_max = 2147483647 net.ipv4.tcp_rmem = 4096 67108864 1073741824 net.ipv4.tcp_wmem = 4096 67108864 1073741824 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096 vm.lowmem_reserve_ratio = 256 256 32 0 0
再度、UDPを試してみます。
$ iperf3 -c 10.0.0.7 -V -u -b 16G
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:12:42 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: 55mtzbjqmr6fy46lxed24cjy7bb4skfujkqh
Target Bitrate: 16000000000
[ 5] local 172.31.8.203 port 40098 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 589 MBytes 4.94 Gbits/sec 426827
[ 5] 1.00-2.00 sec 588 MBytes 4.93 Gbits/sec 425487
[ 5] 2.00-3.00 sec 586 MBytes 4.92 Gbits/sec 424541
[ 5] 3.00-4.00 sec 577 MBytes 4.84 Gbits/sec 417728
[ 5] 4.00-5.00 sec 587 MBytes 4.93 Gbits/sec 425284
[ 5] 5.00-6.00 sec 583 MBytes 4.89 Gbits/sec 422013
[ 5] 6.00-7.00 sec 586 MBytes 4.91 Gbits/sec 424155
[ 5] 7.00-8.00 sec 588 MBytes 4.93 Gbits/sec 425808
[ 5] 8.00-9.00 sec 584 MBytes 4.90 Gbits/sec 422736
[ 5] 9.00-10.00 sec 583 MBytes 4.89 Gbits/sec 422505
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 5.71 GBytes 4.91 Gbits/sec 0.000 ms 0/4237084 (0%) sender
[ 5] 0.00-10.01 sec 2.99 GBytes 2.56 Gbits/sec 0.002 ms 1917865/4133274 (46%) receiver
CPU Utilization: local/sender 98.2% (7.7%u/90.5%s), remote/receiver 99.7% (11.0%u/88.7%s)
iperf Done.
あまり結果は変わりありませんね。
この後、送信側のバッファサイズや-b 4Gに変更したりしましたが、結果は変わりありませんでした。
せっかくバッファサイズを変更したのでTCPでも再度試します。
$ iperf3 -c 10.0.0.7 -V -Z
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:21:39 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: q7iqnyimaj5a2dmnz3u5o4xebfhypr47ntj5
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 36828 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 555 MBytes 4.65 Gbits/sec 0 7.33 MBytes
[ 5] 1.00-2.00 sec 571 MBytes 4.79 Gbits/sec 0 7.33 MBytes
[ 5] 2.00-3.00 sec 572 MBytes 4.80 Gbits/sec 0 7.33 MBytes
[ 5] 3.00-4.00 sec 570 MBytes 4.78 Gbits/sec 0 7.33 MBytes
[ 5] 4.00-5.00 sec 571 MBytes 4.79 Gbits/sec 0 7.33 MBytes
[ 5] 5.00-6.00 sec 570 MBytes 4.78 Gbits/sec 0 7.33 MBytes
[ 5] 6.00-7.00 sec 570 MBytes 4.78 Gbits/sec 0 7.33 MBytes
[ 5] 7.00-8.00 sec 569 MBytes 4.77 Gbits/sec 0 7.33 MBytes
[ 5] 8.00-9.00 sec 562 MBytes 4.71 Gbits/sec 0 7.33 MBytes
[ 5] 9.00-10.00 sec 571 MBytes 4.79 Gbits/sec 135 5.19 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.55 GBytes 4.76 Gbits/sec 135 sender
[ 5] 0.00-10.01 sec 5.53 GBytes 4.75 Gbits/sec receiver
CPU Utilization: local/sender 7.9% (0.1%u/7.8%s), remote/receiver 29.3% (0.9%u/28.4%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
4.75Gbpsと1Gbpsほど転送速度が速くなったように思えます。
TCP 多重度 4
そのまま多重度を増やしていった場合の転送速度の影響を確認しましょう。
まずは多重度4です。
$ iperf3 -c 10.0.0.7 -V -Z -P 4
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:22:45 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: nsxwunjybnhowfxn5roetw3scvxv2jqmqfpf
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 60518 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 60524 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 60534 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 60546 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 4 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 361 MBytes 3.02 Gbits/sec 0 7.94 MBytes
[ 7] 0.00-1.00 sec 361 MBytes 3.03 Gbits/sec 0 8.68 MBytes
[ 9] 0.00-1.00 sec 336 MBytes 2.82 Gbits/sec 0 7.18 MBytes
[ 11] 0.00-1.00 sec 363 MBytes 3.04 Gbits/sec 0 9.83 MBytes
[SUM] 0.00-1.00 sec 1.39 GBytes 11.9 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 354 MBytes 2.97 Gbits/sec 0 8.62 MBytes
[ 7] 1.00-2.00 sec 369 MBytes 3.10 Gbits/sec 0 9.17 MBytes
[ 9] 1.00-2.00 sec 356 MBytes 2.98 Gbits/sec 0 8.29 MBytes
[ 11] 1.00-2.00 sec 368 MBytes 3.08 Gbits/sec 0 9.83 MBytes
[SUM] 1.00-2.00 sec 1.41 GBytes 12.1 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 358 MBytes 3.00 Gbits/sec 0 8.62 MBytes
[ 7] 2.00-3.00 sec 352 MBytes 2.95 Gbits/sec 0 9.63 MBytes
[ 9] 2.00-3.00 sec 345 MBytes 2.89 Gbits/sec 0 8.76 MBytes
[ 11] 2.00-3.00 sec 352 MBytes 2.95 Gbits/sec 0 10.3 MBytes
[SUM] 2.00-3.00 sec 1.37 GBytes 11.8 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 308 MBytes 2.58 Gbits/sec 0 9.17 MBytes
[ 7] 3.00-4.00 sec 329 MBytes 2.76 Gbits/sec 0 9.63 MBytes
[ 9] 3.00-4.00 sec 310 MBytes 2.60 Gbits/sec 0 8.76 MBytes
[ 11] 3.00-4.00 sec 322 MBytes 2.70 Gbits/sec 0 10.3 MBytes
[SUM] 3.00-4.00 sec 1.24 GBytes 10.6 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 350 MBytes 2.94 Gbits/sec 0 9.17 MBytes
[ 7] 4.00-5.00 sec 348 MBytes 2.92 Gbits/sec 0 9.63 MBytes
[ 9] 4.00-5.00 sec 355 MBytes 2.98 Gbits/sec 0 9.26 MBytes
[ 11] 4.00-5.00 sec 360 MBytes 3.02 Gbits/sec 0 10.3 MBytes
[SUM] 4.00-5.00 sec 1.38 GBytes 11.9 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 354 MBytes 2.97 Gbits/sec 0 10.6 MBytes
[ 7] 5.00-6.00 sec 367 MBytes 3.07 Gbits/sec 0 12.3 MBytes
[ 9] 5.00-6.00 sec 352 MBytes 2.95 Gbits/sec 0 11.9 MBytes
[ 11] 5.00-6.00 sec 358 MBytes 3.00 Gbits/sec 0 12.7 MBytes
[SUM] 5.00-6.00 sec 1.40 GBytes 12.0 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 305 MBytes 2.56 Gbits/sec 0 12.5 MBytes
[ 7] 6.00-7.00 sec 380 MBytes 3.19 Gbits/sec 0 14.0 MBytes
[ 9] 6.00-7.00 sec 353 MBytes 2.96 Gbits/sec 0 13.2 MBytes
[ 11] 6.00-7.00 sec 346 MBytes 2.90 Gbits/sec 0 15.5 MBytes
[SUM] 6.00-7.00 sec 1.35 GBytes 11.6 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 346 MBytes 2.90 Gbits/sec 0 13.2 MBytes
[ 7] 7.00-8.00 sec 359 MBytes 3.01 Gbits/sec 0 14.7 MBytes
[ 9] 7.00-8.00 sec 342 MBytes 2.87 Gbits/sec 0 13.9 MBytes
[ 11] 7.00-8.00 sec 361 MBytes 3.03 Gbits/sec 0 16.2 MBytes
[SUM] 7.00-8.00 sec 1.38 GBytes 11.8 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 343 MBytes 2.88 Gbits/sec 0 13.2 MBytes
[ 7] 8.00-9.00 sec 365 MBytes 3.06 Gbits/sec 0 14.7 MBytes
[ 9] 8.00-9.00 sec 356 MBytes 2.99 Gbits/sec 0 13.9 MBytes
[ 11] 8.00-9.00 sec 367 MBytes 3.08 Gbits/sec 0 16.2 MBytes
[SUM] 8.00-9.00 sec 1.40 GBytes 12.0 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 343 MBytes 2.88 Gbits/sec 0 13.2 MBytes
[ 7] 9.00-10.00 sec 355 MBytes 2.98 Gbits/sec 0 14.7 MBytes
[ 9] 9.00-10.00 sec 353 MBytes 2.96 Gbits/sec 0 13.9 MBytes
[ 11] 9.00-10.00 sec 382 MBytes 3.20 Gbits/sec 0 16.2 MBytes
[SUM] 9.00-10.00 sec 1.40 GBytes 12.0 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.34 GBytes 2.87 Gbits/sec 0 sender
[ 5] 0.00-10.02 sec 3.32 GBytes 2.84 Gbits/sec receiver
[ 7] 0.00-10.00 sec 3.50 GBytes 3.01 Gbits/sec 0 sender
[ 7] 0.00-10.02 sec 3.49 GBytes 2.99 Gbits/sec receiver
[ 9] 0.00-10.00 sec 3.38 GBytes 2.90 Gbits/sec 0 sender
[ 9] 0.00-10.02 sec 3.35 GBytes 2.87 Gbits/sec receiver
[ 11] 0.00-10.00 sec 3.49 GBytes 3.00 Gbits/sec 0 sender
[ 11] 0.00-10.02 sec 3.44 GBytes 2.95 Gbits/sec receiver
[SUM] 0.00-10.00 sec 13.7 GBytes 11.8 Gbits/sec 0 sender
[SUM] 0.00-10.02 sec 13.6 GBytes 11.7 Gbits/sec receiver
CPU Utilization: local/sender 15.8% (0.1%u/15.7%s), remote/receiver 93.8% (0.0%u/93.8%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
11.7Gbpsになりました。かなり速いですね。代わりに1接続あたりの転送速度は若干落ちています。
TCP 多重度 8
続いて多重度8です。
$ iperf3 -c 10.0.0.7 -V -Z -P 8
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:23:56 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: wkkwtwi6bpgrhfy3l7gnefeo4dba4z4sal4s
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 44774 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 44778 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 44786 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 44792 connected to 10.0.0.7 port 5201
[ 13] local 172.31.8.203 port 44802 connected to 10.0.0.7 port 5201
[ 15] local 172.31.8.203 port 44816 connected to 10.0.0.7 port 5201
[ 17] local 172.31.8.203 port 44822 connected to 10.0.0.7 port 5201
[ 19] local 172.31.8.203 port 44836 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 8 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 5.75 MBytes 48.2 Mbits/sec 0 372 KBytes
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 161 MBytes 135 Mbits/sec 0 sender
[ 5] 0.00-10.05 sec 161 MBytes 134 Mbits/sec receiver
[ 7] 0.00-10.00 sec 1.99 GBytes 1.71 Gbits/sec 4 sender
[ 7] 0.00-10.05 sec 1.98 GBytes 1.69 Gbits/sec receiver
[ 9] 0.00-10.00 sec 1.53 GBytes 1.32 Gbits/sec 3 sender
[ 9] 0.00-10.05 sec 1.53 GBytes 1.31 Gbits/sec receiver
[ 11] 0.00-10.00 sec 2.03 GBytes 1.74 Gbits/sec 0 sender
[ 11] 0.00-10.05 sec 1.99 GBytes 1.70 Gbits/sec receiver
[ 13] 0.00-10.00 sec 2.10 GBytes 1.80 Gbits/sec 2 sender
[ 13] 0.00-10.05 sec 2.06 GBytes 1.76 Gbits/sec receiver
[ 15] 0.00-10.00 sec 1.92 GBytes 1.65 Gbits/sec 0 sender
[ 15] 0.00-10.05 sec 1.92 GBytes 1.64 Gbits/sec receiver
[ 17] 0.00-10.00 sec 2.03 GBytes 1.74 Gbits/sec 3 sender
[ 17] 0.00-10.05 sec 2.02 GBytes 1.73 Gbits/sec receiver
[ 19] 0.00-10.00 sec 2.03 GBytes 1.74 Gbits/sec 0 sender
[ 19] 0.00-10.05 sec 1.98 GBytes 1.69 Gbits/sec receiver
[SUM] 0.00-10.00 sec 13.8 GBytes 11.8 Gbits/sec 12 sender
[SUM] 0.00-10.05 sec 13.6 GBytes 11.7 Gbits/sec receiver
CPU Utilization: local/sender 26.9% (0.2%u/26.8%s), remote/receiver 94.0% (0.4%u/93.7%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
11.7Gbpsです。多重度4の場合と変わりありませんね。頭打ちのような気がします。
とはいえ、リージョン間のマルチフロートラフィックの場合は帯域幅の最大50％の制限があるところ、11.7Gbps/16Gbpsとかなり検討していると思います。
UDP 多重度 4
UDPでも多重度を変更した場合を確認してみましょう。
$ iperf3 -c 10.0.0.7 -V -u -b 16G -Z -P 4
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:25:12 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: iwtouiaqw4i43rmvjqstwa2z2m72xqsubyln
Target Bitrate: 16000000000
[ 5] local 172.31.8.203 port 41174 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 45334 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 34556 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 35014 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 4 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 183 MBytes 1.53 Gbits/sec 132504
[ 7] 0.00-1.00 sec 183 MBytes 1.53 Gbits/sec 132477
[ 9] 0.00-1.00 sec 184 MBytes 1.54 Gbits/sec 133197
[ 11] 0.00-1.00 sec 183 MBytes 1.54 Gbits/sec 132804
[SUM] 0.00-1.00 sec 733 MBytes 6.15 Gbits/sec 530982
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 183 MBytes 1.53 Gbits/sec 132171
[ 7] 1.00-2.00 sec 184 MBytes 1.54 Gbits/sec 133149
[ 9] 1.00-2.00 sec 183 MBytes 1.54 Gbits/sec 132816
[ 11] 1.00-2.00 sec 184 MBytes 1.54 Gbits/sec 133005
[SUM] 1.00-2.00 sec 733 MBytes 6.15 Gbits/sec 531141
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 184 MBytes 1.54 Gbits/sec 133275
[ 7] 2.00-3.00 sec 184 MBytes 1.55 Gbits/sec 133581
[ 9] 2.00-3.00 sec 185 MBytes 1.55 Gbits/sec 134179
[ 11] 2.00-3.00 sec 185 MBytes 1.55 Gbits/sec 134139
[SUM] 2.00-3.00 sec 739 MBytes 6.20 Gbits/sec 535174
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 184 MBytes 1.54 Gbits/sec 133337
[ 7] 3.00-4.00 sec 185 MBytes 1.55 Gbits/sec 133981
[ 9] 3.00-4.00 sec 184 MBytes 1.55 Gbits/sec 133542
[ 11] 3.00-4.00 sec 184 MBytes 1.54 Gbits/sec 133257
[SUM] 3.00-4.00 sec 738 MBytes 6.19 Gbits/sec 534117
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 184 MBytes 1.54 Gbits/sec 133345
[ 7] 4.00-5.00 sec 184 MBytes 1.55 Gbits/sec 133576
[ 9] 4.00-5.00 sec 184 MBytes 1.55 Gbits/sec 133462
[ 11] 4.00-5.00 sec 185 MBytes 1.56 Gbits/sec 134256
[SUM] 4.00-5.00 sec 738 MBytes 6.19 Gbits/sec 534639
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 184 MBytes 1.54 Gbits/sec 132900
[ 7] 5.00-6.00 sec 185 MBytes 1.55 Gbits/sec 133838
[ 9] 5.00-6.00 sec 185 MBytes 1.55 Gbits/sec 134078
[ 11] 5.00-6.00 sec 185 MBytes 1.55 Gbits/sec 134015
[SUM] 5.00-6.00 sec 739 MBytes 6.20 Gbits/sec 534831
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 184 MBytes 1.54 Gbits/sec 133056
[ 7] 6.00-7.00 sec 185 MBytes 1.55 Gbits/sec 134135
[ 9] 6.00-7.00 sec 185 MBytes 1.55 Gbits/sec 134062
[ 11] 6.00-7.00 sec 184 MBytes 1.55 Gbits/sec 133413
[SUM] 6.00-7.00 sec 738 MBytes 6.19 Gbits/sec 534666
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 185 MBytes 1.55 Gbits/sec 133625
[ 7] 7.00-8.00 sec 185 MBytes 1.55 Gbits/sec 134149
[ 9] 7.00-8.00 sec 185 MBytes 1.55 Gbits/sec 133702
[ 11] 7.00-8.00 sec 185 MBytes 1.55 Gbits/sec 133907
[SUM] 7.00-8.00 sec 739 MBytes 6.20 Gbits/sec 535383
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 185 MBytes 1.55 Gbits/sec 133655
[ 7] 8.00-9.00 sec 185 MBytes 1.55 Gbits/sec 133950
[ 9] 8.00-9.00 sec 185 MBytes 1.55 Gbits/sec 134065
[ 11] 8.00-9.00 sec 185 MBytes 1.55 Gbits/sec 133903
[SUM] 8.00-9.00 sec 740 MBytes 6.20 Gbits/sec 535573
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 184 MBytes 1.54 Gbits/sec 133343
[ 7] 9.00-10.00 sec 185 MBytes 1.55 Gbits/sec 134257
[ 9] 9.00-10.00 sec 186 MBytes 1.56 Gbits/sec 134366
[ 11] 9.00-10.00 sec 185 MBytes 1.55 Gbits/sec 133973
[SUM] 9.00-10.00 sec 740 MBytes 6.21 Gbits/sec 535939
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.80 GBytes 1.54 Gbits/sec 0.000 ms 0/1331211 (0%) sender
[ 5] 0.00-10.03 sec 1.00 GBytes 857 Mbits/sec 0.002 ms 586310/1328129 (44%) receiver
[ 7] 0.00-10.00 sec 1.80 GBytes 1.55 Gbits/sec 0.000 ms 0/1337093 (0%) sender
[ 7] 0.00-10.03 sec 1.00 GBytes 858 Mbits/sec 0.001 ms 582453/1325133 (44%) receiver
[ 9] 0.00-10.00 sec 1.80 GBytes 1.55 Gbits/sec 0.000 ms 0/1337469 (0%) sender
[ 9] 0.00-10.03 sec 1005 MBytes 841 Mbits/sec 0.001 ms 579001/1306840 (44%) receiver
[ 11] 0.00-10.00 sec 1.80 GBytes 1.55 Gbits/sec 0.000 ms 0/1336672 (0%) sender
[ 11] 0.00-10.03 sec 995 MBytes 833 Mbits/sec 0.009 ms 577540/1298296 (44%) receiver
[SUM] 0.00-10.00 sec 7.20 GBytes 6.19 Gbits/sec 0.000 ms 0/5342445 (0%) sender
[SUM] 0.00-10.03 sec 3.96 GBytes 3.39 Gbits/sec 0.003 ms 2325304/5258398 (44%) receiver
CPU Utilization: local/sender 99.3% (7.6%u/91.7%s), remote/receiver 99.4% (5.8%u/93.7%s)
iperf Done.
3.39Gbpsですね。TCPと比べると遅い印象があります。
UDP 多重度 8
続いて多重度8です。
$ iperf3 -c 10.0.0.7 -V -u -b 16G -Z -P 8
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:25:49 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: bbjiqbot63iscimxahiex2454ghxlf4iyib7
Target Bitrate: 16000000000
[ 5] local 172.31.8.203 port 46586 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 41800 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 40048 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 42235 connected to 10.0.0.7 port 5201
[ 13] local 172.31.8.203 port 47925 connected to 10.0.0.7 port 5201
[ 15] local 172.31.8.203 port 53274 connected to 10.0.0.7 port 5201
[ 17] local 172.31.8.203 port 53656 connected to 10.0.0.7 port 5201
[ 19] local 172.31.8.203 port 40819 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 8 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 92.4 MBytes 775 Mbits/sec 66928
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 922 MBytes 774 Mbits/sec 0.000 ms 0/668002 (0%) sender
[ 5] 0.00-10.02 sec 544 MBytes 456 Mbits/sec 0.002 ms 253911/648182 (39%) receiver
[ 7] 0.00-10.00 sec 902 MBytes 757 Mbits/sec 0.000 ms 0/653095 (0%) sender
[ 7] 0.00-10.02 sec 529 MBytes 442 Mbits/sec 0.559 ms 269385/652113 (41%) receiver
[ 9] 0.00-10.00 sec 923 MBytes 774 Mbits/sec 0.000 ms 0/668512 (0%) sender
[ 9] 0.00-10.02 sec 544 MBytes 455 Mbits/sec 0.001 ms 248285/642022 (39%) receiver
[ 11] 0.00-10.00 sec 918 MBytes 770 Mbits/sec 0.000 ms 0/664574 (0%) sender
[ 11] 0.00-10.02 sec 549 MBytes 460 Mbits/sec 0.544 ms 251138/648720 (39%) receiver
[ 13] 0.00-10.00 sec 923 MBytes 775 Mbits/sec 0.000 ms 0/668716 (0%) sender
[ 13] 0.00-10.02 sec 541 MBytes 453 Mbits/sec 0.001 ms 258260/650296 (40%) receiver
[ 15] 0.00-10.00 sec 900 MBytes 755 Mbits/sec 0.000 ms 0/651934 (0%) sender
[ 15] 0.00-10.02 sec 525 MBytes 440 Mbits/sec 0.001 ms 271654/651934 (42%) receiver
[ 17] 0.00-10.00 sec 902 MBytes 757 Mbits/sec 0.000 ms 0/653520 (0%) sender
[ 17] 0.00-10.02 sec 523 MBytes 438 Mbits/sec 0.001 ms 274466/653520 (42%) receiver
[ 19] 0.00-10.00 sec 903 MBytes 757 Mbits/sec 0.000 ms 0/653723 (0%) sender
[ 19] 0.00-10.02 sec 533 MBytes 447 Mbits/sec 0.145 ms 266580/652796 (41%) receiver
[SUM] 0.00-10.00 sec 7.12 GBytes 6.12 Gbits/sec 0.000 ms 0/5282076 (0%) sender
[SUM] 0.00-10.02 sec 4.19 GBytes 3.59 Gbits/sec 0.157 ms 2093679/5199583 (40%) receiver
CPU Utilization: local/sender 99.1% (8.3%u/90.8%s), remote/receiver 98.9% (5.6%u/93.3%s)
iperf Done.
3.59Gbpsです。あまり変わりありませんね。
c6gn.16xlarge の場合
次にc6gn.16xlargeの場合を試してみましょう。
c6gn.16xlargeのネットワーク帯域幅は100Gbpsです。これは期待できますね。
いざ起動
としようとしたところ、大阪リージョン側でキャパシティ不足で立ち上がりませんでした。
c6gn.12xlarge の場合
インスタンスタイプの変更
c6gn.16xlargeが立ち上がらないのであれば、c6gn.12xlargeはどうでしょう。
c6gn.12xlargeのネットワーク帯域幅は75Gbpsです。
こちらは両リージョンとも問題なく立ち上がりました。
EC2インスタンスを停止->起動したのでバッファサイズは元に戻っていました。
$ sudo sysctl -a | grep -e wmem -e rmem -e mtu -e default_qdis net.core.default_qdisc = fq_codel net.core.rmem_default = 212992 net.core.rmem_max = 212992 net.core.wmem_default = 212992 net.core.wmem_max = 212992 net.ipv4.ip_forward_use_pmtu = 0 net.ipv4.ip_no_pmtu_disc = 0 net.ipv4.route.min_pmtu = 552 net.ipv4.route.mtu_expires = 600 net.ipv4.tcp_mtu_probe_floor = 48 net.ipv4.tcp_mtu_probing = 0 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 20480 4194304 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096 net.ipv6.conf.all.accept_ra_mtu = 1 net.ipv6.conf.all.mtu = 1280 net.ipv6.conf.default.accept_ra_mtu = 1 net.ipv6.conf.default.mtu = 1280 net.ipv6.conf.ens5.accept_ra_mtu = 1 net.ipv6.conf.ens5.mtu = 9001 net.ipv6.conf.lo.accept_ra_mtu = 1 net.ipv6.conf.lo.mtu = 65536 net.ipv6.route.mtu_expires = 600 vm.lowmem_reserve_ratio = 256 256 32 0 0 $ ethtool -g ens5 Ring parameters for ens5: Pre-set maximums: RX: 8192 RX Mini: n/a RX Jumbo: n/a TX: 512 Current hardware settings: RX: 1024 RX Mini: n/a RX Jumbo: n/a TX: 512
TCP 多重度 1
TCPの多重度1から試していきます。
$ iperf3 -c 10.0.0.7 -V -Z
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:31:50 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: hjg2vf7634jcqjse5i3ctbi3cdy3uttjdlu5
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 44292 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 316 MBytes 2.65 Gbits/sec 72 4.21 MBytes
[ 5] 1.00-2.00 sec 356 MBytes 2.99 Gbits/sec 0 4.21 MBytes
[ 5] 2.00-3.00 sec 355 MBytes 2.98 Gbits/sec 0 4.21 MBytes
[ 5] 3.00-4.00 sec 357 MBytes 2.99 Gbits/sec 0 4.21 MBytes
[ 5] 4.00-5.00 sec 357 MBytes 2.99 Gbits/sec 0 4.21 MBytes
[ 5] 5.00-6.00 sec 357 MBytes 2.99 Gbits/sec 0 4.21 MBytes
[ 5] 6.00-7.00 sec 355 MBytes 2.98 Gbits/sec 0 4.21 MBytes
[ 5] 7.00-8.00 sec 354 MBytes 2.97 Gbits/sec 0 4.21 MBytes
[ 5] 8.00-9.00 sec 357 MBytes 2.99 Gbits/sec 0 4.21 MBytes
[ 5] 9.00-10.00 sec 351 MBytes 2.94 Gbits/sec 33 3.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.43 GBytes 2.95 Gbits/sec 105 sender
[ 5] 0.00-10.01 sec 3.43 GBytes 2.95 Gbits/sec receiver
CPU Utilization: local/sender 9.1% (0.0%u/9.1%s), remote/receiver 18.7% (0.6%u/18.2%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
2.95 Gbpsです。インスタンスタイプ変更前とあまり変わりありませんね。
TCP 多重度 4
続いて多重度4です。
$ iperf3 -c 10.0.0.7 -V -Z -P 4
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:32:18 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: b774kyln7lkghtnrjzwzyi6scg4jmp4xylpg
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 60466 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 60468 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 60478 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 60482 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 4 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 345 MBytes 2.89 Gbits/sec 0 6.00 MBytes
[ 7] 0.00-1.00 sec 377 MBytes 3.16 Gbits/sec 0 6.01 MBytes
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.56 GBytes 3.06 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 3.56 GBytes 3.05 Gbits/sec receiver
[ 7] 0.00-10.00 sec 3.88 GBytes 3.33 Gbits/sec 0 sender
[ 7] 0.00-10.01 sec 3.88 GBytes 3.33 Gbits/sec receiver
[ 9] 0.00-10.00 sec 3.31 GBytes 2.85 Gbits/sec 0 sender
[ 9] 0.00-10.01 sec 3.31 GBytes 2.84 Gbits/sec receiver
[ 11] 0.00-10.00 sec 3.42 GBytes 2.94 Gbits/sec 0 sender
[ 11] 0.00-10.01 sec 3.42 GBytes 2.93 Gbits/sec receiver
[SUM] 0.00-10.00 sec 14.2 GBytes 12.2 Gbits/sec 0 sender
[SUM] 0.00-10.01 sec 14.2 GBytes 12.2 Gbits/sec receiver
CPU Utilization: local/sender 37.0% (0.2%u/36.8%s), remote/receiver 77.9% (2.8%u/75.1%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
12.2Gbpsです。こちらもインスタンスタイプ変更前とあまり変わりありませんね。
TCP 多重度 8
次に多重度8です。
$ iperf3 -c 10.0.0.7 -V -Z -P 8
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:32:46 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: my4yjwzwcuikfqciajalkm3ij5jlodiqbiw7
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 56626 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 56628 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 56632 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.78 GBytes 3.25 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 3.78 GBytes 3.24 Gbits/sec receiver
[ 7] 0.00-10.00 sec 3.13 GBytes 2.69 Gbits/sec 0 sender
[ 7] 0.00-10.01 sec 3.13 GBytes 2.69 Gbits/sec receiver
[ 9] 0.00-10.00 sec 3.54 GBytes 3.04 Gbits/sec 0 sender
[ 9] 0.00-10.01 sec 3.54 GBytes 3.04 Gbits/sec receiver
[ 11] 0.00-10.00 sec 3.90 GBytes 3.35 Gbits/sec 0 sender
[ 11] 0.00-10.01 sec 3.90 GBytes 3.34 Gbits/sec receiver
[ 13] 0.00-10.00 sec 3.77 GBytes 3.24 Gbits/sec 0 sender
[ 13] 0.00-10.01 sec 3.77 GBytes 3.23 Gbits/sec receiver
[ 15] 0.00-10.00 sec 3.27 GBytes 2.81 Gbits/sec 0 sender
[ 15] 0.00-10.01 sec 3.27 GBytes 2.80 Gbits/sec receiver
[ 17] 0.00-10.00 sec 3.79 GBytes 3.25 Gbits/sec 0 sender
[ 17] 0.00-10.01 sec 3.79 GBytes 3.25 Gbits/sec receiver
[ 19] 0.00-10.00 sec 3.43 GBytes 2.95 Gbits/sec 0 sender
[ 19] 0.00-10.01 sec 3.43 GBytes 2.94 Gbits/sec receiver
[SUM] 0.00-10.00 sec 28.6 GBytes 24.6 Gbits/sec 0 sender
[SUM] 0.00-10.01 sec 28.6 GBytes 24.5 Gbits/sec receiver
CPU Utilization: local/sender 74.2% (0.3%u/73.9%s), remote/receiver 158.3% (4.1%u/154.3%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
24.5Gbpsです。壁を超えた感がありますね。
TCP 多重度 16
さらに増やしていきましょう。多重度16です。
$ iperf3 -c 10.0.0.7 -V -Z -P 16
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:33:12 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: fctf673ypwsjdmwlllj6jrdttdvf72nbqdt5
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 52718 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 52730 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.68 GBytes 3.16 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 3.68 GBytes 3.16 Gbits/sec receiver
[ 7] 0.00-10.00 sec 2.61 GBytes 2.24 Gbits/sec 0 sender
[ 7] 0.00-10.01 sec 2.61 GBytes 2.24 Gbits/sec receiver
[ 9] 0.00-10.00 sec 3.68 GBytes 3.16 Gbits/sec 0 sender
[ 9] 0.00-10.01 sec 3.68 GBytes 3.16 Gbits/sec receiver
[ 11] 0.00-10.00 sec 3.47 GBytes 2.98 Gbits/sec 0 sender
[ 11] 0.00-10.01 sec 3.47 GBytes 2.98 Gbits/sec receiver
[ 13] 0.00-10.00 sec 3.85 GBytes 3.31 Gbits/sec 0 sender
[ 13] 0.00-10.01 sec 3.85 GBytes 3.30 Gbits/sec receiver
[ 15] 0.00-10.00 sec 3.93 GBytes 3.37 Gbits/sec 1413 sender
[ 15] 0.00-10.01 sec 3.93 GBytes 3.37 Gbits/sec receiver
[ 17] 0.00-10.00 sec 2.00 GBytes 1.72 Gbits/sec 623 sender
[ 17] 0.00-10.01 sec 2.00 GBytes 1.72 Gbits/sec receiver
[ 19] 0.00-10.00 sec 3.74 GBytes 3.21 Gbits/sec 0 sender
[ 19] 0.00-10.01 sec 3.73 GBytes 3.21 Gbits/sec receiver
[ 21] 0.00-10.00 sec 2.93 GBytes 2.52 Gbits/sec 0 sender
[ 21] 0.00-10.01 sec 2.93 GBytes 2.51 Gbits/sec receiver
[ 23] 0.00-10.00 sec 3.89 GBytes 3.34 Gbits/sec 0 sender
[ 23] 0.00-10.01 sec 3.89 GBytes 3.34 Gbits/sec receiver
[ 25] 0.00-10.00 sec 3.66 GBytes 3.14 Gbits/sec 0 sender
[ 25] 0.00-10.01 sec 3.66 GBytes 3.14 Gbits/sec receiver
[ 27] 0.00-10.00 sec 3.26 GBytes 2.80 Gbits/sec 0 sender
[ 27] 0.00-10.01 sec 3.26 GBytes 2.80 Gbits/sec receiver
[ 29] 0.00-10.00 sec 3.67 GBytes 3.16 Gbits/sec 0 sender
[ 29] 0.00-10.01 sec 3.67 GBytes 3.15 Gbits/sec receiver
[ 31] 0.00-10.00 sec 3.83 GBytes 3.29 Gbits/sec 0 sender
[ 31] 0.00-10.01 sec 3.83 GBytes 3.29 Gbits/sec receiver
[ 33] 0.00-10.00 sec 3.91 GBytes 3.36 Gbits/sec 0 sender
[ 33] 0.00-10.01 sec 3.91 GBytes 3.36 Gbits/sec receiver
[ 35] 0.00-10.00 sec 3.60 GBytes 3.09 Gbits/sec 0 sender
[ 35] 0.00-10.01 sec 3.60 GBytes 3.09 Gbits/sec receiver
[SUM] 0.00-10.00 sec 55.7 GBytes 47.8 Gbits/sec 2036 sender
[SUM] 0.00-10.01 sec 55.7 GBytes 47.8 Gbits/sec receiver
CPU Utilization: local/sender 139.7% (1.0%u/138.7%s), remote/receiver 322.5% (7.6%u/314.9%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
47.4Gbpsでした。1コネクションの転送速度が安定して3Gbps前後であることからまだまだ余裕はありそうです。
TCP 多重度 32
次に多重度32です。
$ iperf3 -c 10.0.0.7 -V -Z -P 32
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:33:54 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: 2mb67aufkomsevpraug5iue5dhkxdcws4l7i
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 58936 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 58942 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.15 GBytes 1.84 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 2.15 GBytes 1.84 Gbits/sec receiver
[ 7] 0.00-10.00 sec 1.47 GBytes 1.26 Gbits/sec 0 sender
[ 7] 0.00-10.01 sec 1.47 GBytes 1.26 Gbits/sec receiver
[ 9] 0.00-10.00 sec 2.15 GBytes 1.84 Gbits/sec 37 sender
[ 9] 0.00-10.01 sec 2.15 GBytes 1.84 Gbits/sec receiver
[ 11] 0.00-10.00 sec 2.83 GBytes 2.43 Gbits/sec 0 sender
[ 11] 0.00-10.01 sec 2.83 GBytes 2.43 Gbits/sec receiver
[ 13] 0.00-10.00 sec 2.75 GBytes 2.36 Gbits/sec 0 sender
[ 13] 0.00-10.01 sec 2.75 GBytes 2.36 Gbits/sec receiver
[ 15] 0.00-10.00 sec 3.69 GBytes 3.17 Gbits/sec 0 sender
[ 15] 0.00-10.01 sec 3.69 GBytes 3.16 Gbits/sec receiver
[ 17] 0.00-10.00 sec 3.39 GBytes 2.91 Gbits/sec 114 sender
[ 17] 0.00-10.01 sec 3.39 GBytes 2.91 Gbits/sec receiver
[ 19] 0.00-10.00 sec 2.28 GBytes 1.96 Gbits/sec 0 sender
[ 19] 0.00-10.01 sec 2.28 GBytes 1.96 Gbits/sec receiver
[ 21] 0.00-10.00 sec 2.26 GBytes 1.94 Gbits/sec 39 sender
[ 21] 0.00-10.01 sec 2.26 GBytes 1.94 Gbits/sec receiver
[ 23] 0.00-10.00 sec 2.78 GBytes 2.39 Gbits/sec 0 sender
[ 23] 0.00-10.01 sec 2.78 GBytes 2.39 Gbits/sec receiver
[ 25] 0.00-10.00 sec 2.47 GBytes 2.13 Gbits/sec 38 sender
[ 25] 0.00-10.01 sec 2.47 GBytes 2.12 Gbits/sec receiver
[ 27] 0.00-10.00 sec 1.61 GBytes 1.38 Gbits/sec 0 sender
[ 27] 0.00-10.01 sec 1.61 GBytes 1.38 Gbits/sec receiver
[ 29] 0.00-10.00 sec 2.50 GBytes 2.15 Gbits/sec 0 sender
[ 29] 0.00-10.01 sec 2.50 GBytes 2.15 Gbits/sec receiver
[ 31] 0.00-10.00 sec 3.39 GBytes 2.91 Gbits/sec 0 sender
[ 31] 0.00-10.01 sec 3.39 GBytes 2.91 Gbits/sec receiver
[ 33] 0.00-10.00 sec 1018 MBytes 854 Mbits/sec 0 sender
[ 33] 0.00-10.01 sec 1016 MBytes 851 Mbits/sec receiver
[ 35] 0.00-10.00 sec 1.98 GBytes 1.70 Gbits/sec 0 sender
[ 35] 0.00-10.01 sec 1.98 GBytes 1.70 Gbits/sec receiver
[ 37] 0.00-10.00 sec 3.04 GBytes 2.61 Gbits/sec 0 sender
[ 37] 0.00-10.01 sec 3.04 GBytes 2.61 Gbits/sec receiver
[ 39] 0.00-10.00 sec 2.84 GBytes 2.44 Gbits/sec 0 sender
[ 39] 0.00-10.01 sec 2.84 GBytes 2.44 Gbits/sec receiver
[ 41] 0.00-10.00 sec 3.24 GBytes 2.78 Gbits/sec 0 sender
[ 41] 0.00-10.01 sec 3.24 GBytes 2.78 Gbits/sec receiver
[ 43] 0.00-10.00 sec 3.26 GBytes 2.80 Gbits/sec 0 sender
[ 43] 0.00-10.01 sec 3.26 GBytes 2.80 Gbits/sec receiver
[ 45] 0.00-10.00 sec 3.09 GBytes 2.65 Gbits/sec 0 sender
[ 45] 0.00-10.01 sec 3.09 GBytes 2.65 Gbits/sec receiver
[ 47] 0.00-10.00 sec 3.17 GBytes 2.72 Gbits/sec 0 sender
[ 47] 0.00-10.01 sec 3.17 GBytes 2.72 Gbits/sec receiver
[ 49] 0.00-10.00 sec 1.99 GBytes 1.71 Gbits/sec 59 sender
[ 49] 0.00-10.01 sec 1.99 GBytes 1.71 Gbits/sec receiver
[ 51] 0.00-10.00 sec 2.56 GBytes 2.20 Gbits/sec 0 sender
[ 51] 0.00-10.01 sec 2.56 GBytes 2.19 Gbits/sec receiver
[ 53] 0.00-10.00 sec 2.19 GBytes 1.88 Gbits/sec 41 sender
[ 53] 0.00-10.01 sec 2.19 GBytes 1.88 Gbits/sec receiver
[ 55] 0.00-10.00 sec 2.43 GBytes 2.09 Gbits/sec 0 sender
[ 55] 0.00-10.01 sec 2.43 GBytes 2.09 Gbits/sec receiver
[ 57] 0.00-10.00 sec 2.78 GBytes 2.39 Gbits/sec 0 sender
[ 57] 0.00-10.01 sec 2.78 GBytes 2.39 Gbits/sec receiver
[ 59] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec 0 sender
[ 59] 0.00-10.01 sec 1.91 GBytes 1.64 Gbits/sec receiver
[ 61] 0.00-10.00 sec 1.47 GBytes 1.26 Gbits/sec 0 sender
[ 61] 0.00-10.01 sec 1.47 GBytes 1.26 Gbits/sec receiver
[ 63] 0.00-10.00 sec 1.50 GBytes 1.29 Gbits/sec 0 sender
[ 63] 0.00-10.01 sec 1.50 GBytes 1.29 Gbits/sec receiver
[ 65] 0.00-10.00 sec 1.62 GBytes 1.39 Gbits/sec 0 sender
[ 65] 0.00-10.01 sec 1.62 GBytes 1.39 Gbits/sec receiver
[ 67] 0.00-10.00 sec 1.62 GBytes 1.39 Gbits/sec 0 sender
[ 67] 0.00-10.01 sec 1.62 GBytes 1.39 Gbits/sec receiver
[SUM] 0.00-10.00 sec 77.4 GBytes 66.5 Gbits/sec 328 sender
[SUM] 0.00-10.01 sec 77.4 GBytes 66.4 Gbits/sec receiver
CPU Utilization: local/sender 110.9% (0.8%u/110.1%s), remote/receiver 460.0% (10.9%u/449.1%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
66.4Gbpsでした。1コネクションあたりの転送速度が少し落ちてきましたね。
TCP 多重度 64
多重度64です。
$ iperf3 -c 10.0.0.7 -V -Z -P 64
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:34:25 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: nbaiokldyhhkrreqfbxpbdli4ubx6kr3e6s2
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 60776 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 60780 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 60782 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 60788 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 1.18 GBytes 1.01 Gbits/sec 128 sender
[ 5] 0.00-10.01 sec 1.17 GBytes 1.01 Gbits/sec receiver
[ 7] 0.00-10.01 sec 1022 MBytes 857 Mbits/sec 180 sender
[ 7] 0.00-10.01 sec 1020 MBytes 854 Mbits/sec receiver
[ 9] 0.00-10.01 sec 866 MBytes 726 Mbits/sec 190 sender
[ 9] 0.00-10.01 sec 864 MBytes 724 Mbits/sec receiver
.
.
(中略)
.
.
[119] 0.00-10.01 sec 762 MBytes 638 Mbits/sec receiver
[121] 0.00-10.01 sec 920 MBytes 771 Mbits/sec 151 sender
[121] 0.00-10.01 sec 918 MBytes 769 Mbits/sec receiver
[123] 0.00-10.01 sec 706 MBytes 592 Mbits/sec 0 sender
[123] 0.00-10.01 sec 704 MBytes 590 Mbits/sec receiver
[125] 0.00-10.01 sec 1.51 GBytes 1.30 Gbits/sec 0 sender
[125] 0.00-10.01 sec 1.51 GBytes 1.29 Gbits/sec receiver
[127] 0.00-10.01 sec 2.06 GBytes 1.77 Gbits/sec 0 sender
[127] 0.00-10.01 sec 2.06 GBytes 1.77 Gbits/sec receiver
[129] 0.00-10.01 sec 1.58 GBytes 1.35 Gbits/sec 54 sender
[129] 0.00-10.01 sec 1.57 GBytes 1.35 Gbits/sec receiver
[131] 0.00-10.01 sec 1.04 GBytes 889 Mbits/sec 18 sender
[131] 0.00-10.01 sec 1.03 GBytes 886 Mbits/sec receiver
[SUM] 0.00-10.01 sec 82.9 GBytes 71.2 Gbits/sec 6654 sender
[SUM] 0.00-10.01 sec 82.8 GBytes 71.0 Gbits/sec receiver
CPU Utilization: local/sender 87.2% (0.6%u/86.6%s), remote/receiver 519.9% (11.9%u/508.1%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
71.0Gbpsでした。1コネクションあたりの転送速度からしてここらが限界でしょう。
異なるリージョンであるにも関わらず70Gbpsで転送できるのは素晴らしいですね。
また、再送されたTCPセグメントが6,654とかなり増えてきました。本来であれば帯域幅を-b 2Gなどある程度絞ってあげた方がいいかもしれないですね。
TCP 多重度 96
最後に多重度96です。
$ iperf3 -c 10.0.0.7 -V -Z -P 96
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:35:32 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: hnro4ufbbc6yvv5apahsgo64dsb3c45xzano
TCP MSS: 1448 (default)
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 710 MBytes 595 Mbits/sec 129 sender
[ 5] 0.00-10.01 sec 708 MBytes 593 Mbits/sec receiver
.
.
(中略)
.
.
[195] 0.00-10.01 sec 1.17 GBytes 1.01 Gbits/sec 90 sender
[195] 0.00-10.01 sec 1.17 GBytes 1.01 Gbits/sec receiver
[SUM] 0.00-10.01 sec 83.2 GBytes 71.4 Gbits/sec 13217 sender
[SUM] 0.00-10.01 sec 83.0 GBytes 71.2 Gbits/sec receiver
CPU Utilization: local/sender 81.6% (1.2%u/80.4%s), remote/receiver 540.9% (13.3%u/527.6%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
71.2Gbpsでした。多重度64Gbpsで頭打ちだったようですね。
UDP 多重度 1
UDPでも多重度を変えて試してみます。
$ iperf3 -c 10.0.0.7 -V -Z -u -b 16G
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:37:15 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: 5bboi4prm4x7fc3gg6nl6of2t4fbjbir64wp
Target Bitrate: 16000000000
[ 5] local 172.31.8.203 port 52388 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 580 MBytes 4.87 Gbits/sec 420136
[ 5] 1.00-2.00 sec 579 MBytes 4.86 Gbits/sec 419551
[ 5] 2.00-3.00 sec 579 MBytes 4.86 Gbits/sec 419543
[ 5] 3.00-4.00 sec 579 MBytes 4.86 Gbits/sec 419566
[ 5] 4.00-5.00 sec 579 MBytes 4.86 Gbits/sec 419553
[ 5] 5.00-6.00 sec 579 MBytes 4.86 Gbits/sec 419559
[ 5] 6.00-7.00 sec 579 MBytes 4.86 Gbits/sec 419544
[ 5] 7.00-8.00 sec 579 MBytes 4.86 Gbits/sec 419548
[ 5] 8.00-9.00 sec 579 MBytes 4.86 Gbits/sec 419572
[ 5] 9.00-10.00 sec 579 MBytes 4.86 Gbits/sec 419563
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 5.66 GBytes 4.86 Gbits/sec 0.000 ms 0/4196135 (0%) sender
[ 5] 0.00-10.01 sec 5.54 GBytes 4.75 Gbits/sec 0.001 ms 87853/4195512 (2.1%) receiver
CPU Utilization: local/sender 72.4% (6.3%u/66.0%s), remote/receiver 199.6% (20.0%u/179.6%s)
iperf Done.
4.75Gbpsでした。メモリサイズが増えたからかロストしたデータグラムも少なそうです。
UDP 多重度 4
次に多重度4です。
$ iperf3 -c 10.0.0.7 -V -Z -u -b 16G -P 4
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:37:53 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: ijqxkpaz6xvxkhtqy6idzlj5wt3nv6huafgt
Target Bitrate: 16000000000
[ 5] local 172.31.8.203 port 41792 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 33319 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 32915 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 41212 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 4 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 580 MBytes 4.87 Gbits/sec 420035
[ 7] 0.00-1.00 sec 544 MBytes 4.57 Gbits/sec 394207
[ 9] 0.00-1.00 sec 580 MBytes 4.86 Gbits/sec 420002
[ 11] 0.00-1.00 sec 580 MBytes 4.87 Gbits/sec 420047
[SUM] 0.00-1.00 sec 2.23 GBytes 19.2 Gbits/sec 1654291
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 579 MBytes 4.86 Gbits/sec 419568
[ 7] 1.00-2.00 sec 549 MBytes 4.61 Gbits/sec 397590
[ 9] 1.00-2.00 sec 579 MBytes 4.86 Gbits/sec 419531
[ 11] 1.00-2.00 sec 579 MBytes 4.86 Gbits/sec 419559
[SUM] 1.00-2.00 sec 2.23 GBytes 19.2 Gbits/sec 1656248
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 579 MBytes 4.86 Gbits/sec 419525
[ 7] 2.00-3.00 sec 550 MBytes 4.61 Gbits/sec 398189
[ 9] 2.00-3.00 sec 579 MBytes 4.86 Gbits/sec 419576
[ 11] 2.00-3.00 sec 579 MBytes 4.86 Gbits/sec 419555
[SUM] 2.00-3.00 sec 2.23 GBytes 19.2 Gbits/sec 1656845
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 579 MBytes 4.86 Gbits/sec 419574
[ 7] 3.00-4.00 sec 547 MBytes 4.59 Gbits/sec 396229
[ 9] 3.00-4.00 sec 579 MBytes 4.86 Gbits/sec 419555
[ 11] 3.00-4.00 sec 579 MBytes 4.86 Gbits/sec 419566
[SUM] 3.00-4.00 sec 2.23 GBytes 19.2 Gbits/sec 1654924
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 579 MBytes 4.86 Gbits/sec 419542
[ 7] 4.00-5.00 sec 546 MBytes 4.58 Gbits/sec 395182
[ 9] 4.00-5.00 sec 579 MBytes 4.86 Gbits/sec 419551
[ 11] 4.00-5.00 sec 579 MBytes 4.86 Gbits/sec 419544
[SUM] 4.00-5.00 sec 2.23 GBytes 19.2 Gbits/sec 1653819
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 579 MBytes 4.86 Gbits/sec 419548
[ 7] 5.00-6.00 sec 542 MBytes 4.55 Gbits/sec 392605
[ 9] 5.00-6.00 sec 579 MBytes 4.86 Gbits/sec 419579
[ 11] 5.00-6.00 sec 579 MBytes 4.86 Gbits/sec 419563
[SUM] 5.00-6.00 sec 2.23 GBytes 19.1 Gbits/sec 1651295
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 579 MBytes 4.86 Gbits/sec 419579
[ 7] 6.00-7.00 sec 550 MBytes 4.61 Gbits/sec 398216
[ 9] 6.00-7.00 sec 579 MBytes 4.86 Gbits/sec 419551
[ 11] 6.00-7.00 sec 579 MBytes 4.86 Gbits/sec 419561
[SUM] 6.00-7.00 sec 2.23 GBytes 19.2 Gbits/sec 1656907
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 579 MBytes 4.86 Gbits/sec 419550
[ 7] 7.00-8.00 sec 546 MBytes 4.58 Gbits/sec 395661
[ 9] 7.00-8.00 sec 579 MBytes 4.86 Gbits/sec 419549
[ 11] 7.00-8.00 sec 579 MBytes 4.86 Gbits/sec 419556
[SUM] 7.00-8.00 sec 2.23 GBytes 19.2 Gbits/sec 1654316
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 579 MBytes 4.86 Gbits/sec 419569
[ 7] 8.00-9.00 sec 546 MBytes 4.58 Gbits/sec 395390
[ 9] 8.00-9.00 sec 579 MBytes 4.86 Gbits/sec 419548
[ 11] 8.00-9.00 sec 579 MBytes 4.86 Gbits/sec 419545
[SUM] 8.00-9.00 sec 2.23 GBytes 19.2 Gbits/sec 1654052
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 579 MBytes 4.86 Gbits/sec 419532
[ 7] 9.00-10.00 sec 542 MBytes 4.55 Gbits/sec 392531
[ 9] 9.00-10.00 sec 579 MBytes 4.86 Gbits/sec 419527
[ 11] 9.00-10.00 sec 579 MBytes 4.86 Gbits/sec 419563
[SUM] 9.00-10.00 sec 2.23 GBytes 19.1 Gbits/sec 1651153
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 5.66 GBytes 4.86 Gbits/sec 0.000 ms 0/4196022 (0%) sender
[ 5] 0.00-10.01 sec 5.65 GBytes 4.85 Gbits/sec 0.001 ms 3557/4195779 (0.085%) receiver
[ 7] 0.00-10.00 sec 5.33 GBytes 4.58 Gbits/sec 0.000 ms 0/3955800 (0%) sender
[ 7] 0.00-10.01 sec 5.33 GBytes 4.58 Gbits/sec 0.001 ms 456/3955798 (0.012%) receiver
[ 9] 0.00-10.00 sec 5.66 GBytes 4.86 Gbits/sec 0.000 ms 0/4195969 (0%) sender
[ 9] 0.00-10.01 sec 5.66 GBytes 4.86 Gbits/sec 0.002 ms 0/4195969 (0%) receiver
[ 11] 0.00-10.00 sec 5.66 GBytes 4.86 Gbits/sec 0.000 ms 0/4196059 (0%) sender
[ 11] 0.00-10.01 sec 5.65 GBytes 4.85 Gbits/sec 0.001 ms 9431/4196059 (0.22%) receiver
[SUM] 0.00-10.00 sec 22.3 GBytes 19.2 Gbits/sec 0.000 ms 0/16543850 (0%) sender
[SUM] 0.00-10.01 sec 22.3 GBytes 19.1 Gbits/sec 0.001 ms 13444/16543605 (0.081%) receiver
CPU Utilization: local/sender 324.9% (34.4%u/290.5%s), remote/receiver 439.4% (51.9%u/387.5%s)
iperf Done.
19.1Gbpsでした。TCPの多重度4と比較すると多少速いですね。
バッファサイズ変更
Fasterdata Knowledge BaseのLinux Tuningを参考に東京と大阪どちらのEC2インスタンスもバッファサイズを変更してみます。
$ sudo sysctl -w net.ipv4.tcp_wmem="4096 67108864 1073741824" $ sudo sysctl -w net.ipv4.tcp_rmem="4096 67108864 1073741824" $ sudo sysctl -w net.core.wmem_default=2147483647 $ sudo sysctl -w net.core.rmem_default=2147483647 $ sudo sysctl -w net.core.wmem_max=2147483647 $ sudo sysctl -w net.core.rmem_max=2147483647 $ sudo sysctl -a | grep -e wmem -e rmem net.core.rmem_default = 2147483647 net.core.rmem_max = 2147483647 net.core.wmem_default = 2147483647 net.core.wmem_max = 2147483647 net.ipv4.tcp_rmem = 4096 67108864 1073741824 net.ipv4.tcp_wmem = 4096 67108864 1073741824 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096 vm.lowmem_reserve_ratio = 256 256 32 0 0
TCP 多重度 1
再度TCPから確認していきます。
$ iperf3 -c 10.0.0.7 -V -Z
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:40:12 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: yho6tfdmzv4552akmdryyh2typziyvrzu6ow
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 57492 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 580 MBytes 4.86 Gbits/sec 0 6.85 MBytes
[ 5] 1.00-2.00 sec 585 MBytes 4.91 Gbits/sec 0 6.85 MBytes
[ 5] 2.00-3.00 sec 563 MBytes 4.72 Gbits/sec 0 6.85 MBytes
[ 5] 3.00-4.00 sec 562 MBytes 4.72 Gbits/sec 0 6.85 MBytes
[ 5] 4.00-5.00 sec 585 MBytes 4.91 Gbits/sec 0 6.85 MBytes
[ 5] 5.00-6.00 sec 562 MBytes 4.72 Gbits/sec 0 6.85 MBytes
[ 5] 6.00-7.00 sec 562 MBytes 4.72 Gbits/sec 0 6.85 MBytes
[ 5] 7.00-8.00 sec 585 MBytes 4.90 Gbits/sec 0 6.85 MBytes
[ 5] 8.00-9.00 sec 562 MBytes 4.72 Gbits/sec 0 6.85 MBytes
[ 5] 9.00-10.00 sec 562 MBytes 4.72 Gbits/sec 0 6.85 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.58 GBytes 4.79 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 5.54 GBytes 4.75 Gbits/sec receiver
CPU Utilization: local/sender 6.9% (0.1%u/6.8%s), remote/receiver 28.9% (1.0%u/27.9%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
4.75Gbpsとシングルフロートラフィックの上限5Gbps近く出ていますね。
TCP 多重度 16
順当に1コネクションあたりの転送速度が増していそうなので、多重度を一気に16に増やしてみます。
$ iperf3 -c 10.0.0.7 -V -Z -P 16
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:40:56 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: deopdqcynvkgo3yosp6zhhqxqmckrz3ani76
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 51372 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 51382 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 51386 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.62 GBytes 3.97 Gbits/sec 497 sender
[ 5] 0.00-10.01 sec 4.58 GBytes 3.93 Gbits/sec receiver
[ 7] 0.00-10.00 sec 3.19 GBytes 2.74 Gbits/sec 0 sender
[ 7] 0.00-10.01 sec 3.14 GBytes 2.70 Gbits/sec receiver
[ 9] 0.00-10.00 sec 4.44 GBytes 3.81 Gbits/sec 256 sender
[ 9] 0.00-10.01 sec 4.40 GBytes 3.78 Gbits/sec receiver
[ 11] 0.00-10.00 sec 4.56 GBytes 3.91 Gbits/sec 478 sender
[ 11] 0.00-10.01 sec 4.52 GBytes 3.88 Gbits/sec receiver
[ 13] 0.00-10.00 sec 4.84 GBytes 4.16 Gbits/sec 226 sender
[ 13] 0.00-10.01 sec 4.79 GBytes 4.11 Gbits/sec receiver
[ 15] 0.00-10.00 sec 4.97 GBytes 4.27 Gbits/sec 268 sender
[ 15] 0.00-10.01 sec 4.92 GBytes 4.22 Gbits/sec receiver
[ 17] 0.00-10.00 sec 5.15 GBytes 4.43 Gbits/sec 258 sender
[ 17] 0.00-10.01 sec 5.10 GBytes 4.37 Gbits/sec receiver
[ 19] 0.00-10.00 sec 4.85 GBytes 4.17 Gbits/sec 180 sender
[ 19] 0.00-10.01 sec 4.81 GBytes 4.13 Gbits/sec receiver
[ 21] 0.00-10.00 sec 2.64 GBytes 2.26 Gbits/sec 61 sender
[ 21] 0.00-10.01 sec 2.59 GBytes 2.23 Gbits/sec receiver
[ 23] 0.00-10.00 sec 2.53 GBytes 2.18 Gbits/sec 75 sender
[ 23] 0.00-10.01 sec 2.48 GBytes 2.13 Gbits/sec receiver
[ 25] 0.00-10.00 sec 4.27 GBytes 3.67 Gbits/sec 170 sender
[ 25] 0.00-10.01 sec 4.22 GBytes 3.62 Gbits/sec receiver
[ 27] 0.00-10.00 sec 4.76 GBytes 4.08 Gbits/sec 52 sender
[ 27] 0.00-10.01 sec 4.70 GBytes 4.04 Gbits/sec receiver
[ 29] 0.00-10.00 sec 5.39 GBytes 4.63 Gbits/sec 46 sender
[ 29] 0.00-10.01 sec 5.36 GBytes 4.60 Gbits/sec receiver
[ 31] 0.00-10.00 sec 3.39 GBytes 2.91 Gbits/sec 341 sender
[ 31] 0.00-10.01 sec 3.35 GBytes 2.87 Gbits/sec receiver
[ 33] 0.00-10.00 sec 2.45 GBytes 2.10 Gbits/sec 0 sender
[ 33] 0.00-10.01 sec 2.40 GBytes 2.06 Gbits/sec receiver
[ 35] 0.00-10.00 sec 4.78 GBytes 4.11 Gbits/sec 504 sender
[ 35] 0.00-10.01 sec 4.74 GBytes 4.06 Gbits/sec receiver
[SUM] 0.00-10.00 sec 66.8 GBytes 57.4 Gbits/sec 3412 sender
[SUM] 0.00-10.01 sec 66.1 GBytes 56.7 Gbits/sec receiver
CPU Utilization: local/sender 73.7% (0.8%u/72.9%s), remote/receiver 334.1% (7.4%u/326.7%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
56.7Gbpsでした。バッファサイズを変更する前は47.4Gbpsだったので転送速度が改善されていそうです。
TCP 多重度 32
次に多重度32です。
$ iperf3 -c 10.0.0.7 -V -Z -P 32
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:42:08 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: 4mydnhhyspqsnx3pqqqbe5xdn42hw22ujvd6
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 41494 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 41504 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.48 GBytes 2.13 Gbits/sec 55 sender
[ 5] 0.00-10.01 sec 2.43 GBytes 2.08 Gbits/sec receiver
[ 7] 0.00-10.00 sec 2.37 GBytes 2.03 Gbits/sec 0 sender
[ 7] 0.00-10.01 sec 2.33 GBytes 2.00 Gbits/sec receiver
[ 9] 0.00-10.00 sec 3.43 GBytes 2.94 Gbits/sec 82 sender
[ 9] 0.00-10.01 sec 3.39 GBytes 2.91 Gbits/sec receiver
.
.
(中略)
.
.
[ 59] 0.00-10.00 sec 1.97 GBytes 1.69 Gbits/sec 65 sender
[ 59] 0.00-10.01 sec 1.91 GBytes 1.64 Gbits/sec receiver
[ 61] 0.00-10.00 sec 1.95 GBytes 1.68 Gbits/sec 77 sender
[ 61] 0.00-10.01 sec 1.91 GBytes 1.64 Gbits/sec receiver
[ 63] 0.00-10.00 sec 2.75 GBytes 2.36 Gbits/sec 131 sender
[ 63] 0.00-10.01 sec 2.70 GBytes 2.32 Gbits/sec receiver
[ 65] 0.00-10.00 sec 2.20 GBytes 1.89 Gbits/sec 8 sender
[ 65] 0.00-10.01 sec 2.15 GBytes 1.85 Gbits/sec receiver
[ 67] 0.00-10.00 sec 2.19 GBytes 1.88 Gbits/sec 3 sender
[ 67] 0.00-10.01 sec 2.15 GBytes 1.84 Gbits/sec receiver
[SUM] 0.00-10.00 sec 82.9 GBytes 71.2 Gbits/sec 2501 sender
[SUM] 0.00-10.01 sec 81.4 GBytes 69.8 Gbits/sec receiver
CPU Utilization: local/sender 87.8% (0.8%u/87.0%s), remote/receiver 422.3% (12.5%u/409.8%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
69.8Gbpsでした。
TCP 多重度 64
続いて多重度64です。
$ iperf3 -c 10.0.0.7 -V -Z -P 64
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:42:47 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: q2rmrlwcrb46ikv6fpo5f4zlniitn37lvr5y
TCP MSS: 1448 (default)
[ 5] local 172.31.8.203 port 48948 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 48956 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 1.06 GBytes 909 Mbits/sec 64 sender
[ 5] 0.00-10.02 sec 1.00 GBytes 858 Mbits/sec receiver
[ 7] 0.00-10.01 sec 995 MBytes 834 Mbits/sec 12 sender
[ 7] 0.00-10.02 sec 952 MBytes 798 Mbits/sec receiver
[ 9] 0.00-10.01 sec 2.07 GBytes 1.78 Gbits/sec 52 sender
[ 9] 0.00-10.02 sec 2.02 GBytes 1.74 Gbits/sec receiver
[ 11] 0.00-10.01 sec 1.14 GBytes 975 Mbits/sec 230 sender
[ 11] 0.00-10.02 sec 1.09 GBytes 933 Mbits/sec receiver
[ 13] 0.00-10.01 sec 1.20 GBytes 1.03 Gbits/sec 140 sender
[ 13] 0.00-10.02 sec 1.15 GBytes 983 Mbits/sec receiver
.
.
(中略)
.
.
[123] 0.00-10.01 sec 779 MBytes 653 Mbits/sec 112 sender
[123] 0.00-10.02 sec 726 MBytes 608 Mbits/sec receiver
[125] 0.00-10.01 sec 1.26 GBytes 1.09 Gbits/sec 194 sender
[125] 0.00-10.02 sec 1.21 GBytes 1.04 Gbits/sec receiver
[127] 0.00-10.01 sec 1.77 GBytes 1.52 Gbits/sec 24 sender
[127] 0.00-10.02 sec 1.72 GBytes 1.47 Gbits/sec receiver
[129] 0.00-10.01 sec 886 MBytes 743 Mbits/sec 32 sender
[129] 0.00-10.02 sec 830 MBytes 695 Mbits/sec receiver
[131] 0.00-10.01 sec 971 MBytes 814 Mbits/sec 15 sender
[131] 0.00-10.02 sec 918 MBytes 769 Mbits/sec receiver
[SUM] 0.00-10.01 sec 86.1 GBytes 73.9 Gbits/sec 7104 sender
[SUM] 0.00-10.02 sec 83.0 GBytes 71.1 Gbits/sec receiver
CPU Utilization: local/sender 90.0% (1.1%u/88.9%s), remote/receiver 496.4% (14.3%u/482.1%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
71.1Gbpsでした。バッファサイズ変更前とあまり変わりありませんね。
UDP 多重度 16
TCPでインスタンス上限の75Gbpsに近い値を出せたので満足ですが、UDPでも何回か試してみます。
まず、多重度16です。
$ iperf3 -c 10.0.0.7 -V -Z -u -b 16G -P 16
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:43:51 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: 22yrdubjcyytsaw6fonholveldzfke5nz3ns
Target Bitrate: 16000000000
[ 5] local 172.31.8.203 port 39826 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 55985 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 35873 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 4.30 GBytes 3.70 Gbits/sec 0.000 ms 0/3191957 (0%) sender
[ 5] 0.00-10.01 sec 4.30 GBytes 3.69 Gbits/sec 0.002 ms 0/3191957 (0%) receiver
[ 7] 0.00-10.00 sec 3.11 GBytes 2.68 Gbits/sec 0.000 ms 0/2309671 (0%) sender
[ 7] 0.00-10.01 sec 1.48 GBytes 1.27 Gbits/sec 0.005 ms 1206485/2303519 (52%) receiver
[ 9] 0.00-10.00 sec 3.69 GBytes 3.17 Gbits/sec 0.000 ms 0/2738909 (0%) sender
[ 9] 0.00-10.01 sec 1.59 GBytes 1.36 Gbits/sec 0.007 ms 1555112/2733844 (57%) receiver
[ 11] 0.00-10.00 sec 4.07 GBytes 3.49 Gbits/sec 0.000 ms 0/3015177 (0%) sender
[ 11] 0.00-10.01 sec 1.36 GBytes 1.17 Gbits/sec 0.007 ms 1994688/3003591 (66%) receiver
[ 13] 0.00-10.00 sec 3.46 GBytes 2.97 Gbits/sec 0.000 ms 0/2564061 (0%) sender
[ 13] 0.00-10.01 sec 1.62 GBytes 1.39 Gbits/sec 0.007 ms 1358020/2559007 (53%) receiver
[ 15] 0.00-10.00 sec 4.49 GBytes 3.86 Gbits/sec 0.000 ms 0/3332373 (0%) sender
[ 15] 0.00-10.01 sec 4.49 GBytes 3.86 Gbits/sec 0.003 ms 137/3332373 (0.0041%) receiver
[ 17] 0.00-10.00 sec 4.16 GBytes 3.57 Gbits/sec 0.000 ms 0/3082430 (0%) sender
[ 17] 0.00-10.01 sec 1.83 GBytes 1.57 Gbits/sec 0.002 ms 1712705/3071249 (56%) receiver
[ 19] 0.00-10.00 sec 3.33 GBytes 2.86 Gbits/sec 0.000 ms 0/2470818 (0%) sender
[ 19] 0.00-10.01 sec 2.64 GBytes 2.26 Gbits/sec 0.003 ms 507865/2464426 (21%) receiver
[ 21] 0.00-10.00 sec 4.07 GBytes 3.50 Gbits/sec 0.000 ms 0/3017352 (0%) sender
[ 21] 0.00-10.01 sec 1.38 GBytes 1.18 Gbits/sec 0.012 ms 1984736/3005727 (66%) receiver
[ 23] 0.00-10.00 sec 4.60 GBytes 3.95 Gbits/sec 0.000 ms 0/3410073 (0%) sender
[ 23] 0.00-10.01 sec 4.60 GBytes 3.95 Gbits/sec 0.002 ms 0/3410073 (0%) receiver
[ 25] 0.00-10.00 sec 4.06 GBytes 3.49 Gbits/sec 0.000 ms 0/3009308 (0%) sender
[ 25] 0.00-10.01 sec 4.06 GBytes 3.48 Gbits/sec 0.002 ms 0/3009308 (0%) receiver
[ 27] 0.00-10.00 sec 4.29 GBytes 3.69 Gbits/sec 0.000 ms 0/3182991 (0%) sender
[ 27] 0.00-10.01 sec 3.02 GBytes 2.59 Gbits/sec 0.002 ms 933114/3171473 (29%) receiver
[ 29] 0.00-10.00 sec 4.47 GBytes 3.84 Gbits/sec 0.000 ms 0/3310993 (0%) sender
[ 29] 0.00-10.01 sec 4.47 GBytes 3.83 Gbits/sec 0.003 ms 0/3310993 (0%) receiver
[ 31] 0.00-10.00 sec 3.62 GBytes 3.11 Gbits/sec 0.000 ms 0/2687322 (0%) sender
[ 31] 0.00-10.01 sec 3.62 GBytes 3.11 Gbits/sec 0.003 ms 0/2687294 (0%) receiver
[ 33] 0.00-10.00 sec 4.03 GBytes 3.46 Gbits/sec 0.000 ms 0/2991237 (0%) sender
[ 33] 0.00-10.01 sec 2.91 GBytes 2.50 Gbits/sec 0.005 ms 827909/2986253 (28%) receiver
[ 35] 0.00-10.00 sec 3.84 GBytes 3.29 Gbits/sec 0.000 ms 0/2844036 (0%) sender
[ 35] 0.00-10.01 sec 1.44 GBytes 1.24 Gbits/sec 0.015 ms 1768709/2837588 (62%) receiver
[SUM] 0.00-10.00 sec 63.6 GBytes 54.6 Gbits/sec 0.000 ms 0/47158708 (0%) sender
[SUM] 0.00-10.01 sec 44.8 GBytes 38.5 Gbits/sec 0.005 ms 13849480/47078675 (29%) receiver
CPU Utilization: local/sender 1579.0% (112.9%u/1466.1%s), remote/receiver 828.9% (95.1%u/733.8%s)
iperf Done.
38.5Gbpsでした。
UDP 多重度 64
最後に多重度64です。
$ iperf3 -c 10.0.0.7 -V -Z -u -b 16G -P 64
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:44:37 GMT
Connecting to host 10.0.0.7, port 5201
Cookie: ocekrkbm2w6puttizmq3rzlrxbtrjs4y7f6v
Target Bitrate: 16000000000
[ 5] local 172.31.8.203 port 52683 connected to 10.0.0.7 port 5201
[ 7] local 172.31.8.203 port 43005 connected to 10.0.0.7 port 5201
[ 9] local 172.31.8.203 port 47483 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 999 MBytes 838 Mbits/sec 0.000 ms 0/723751 (0%) sender
[ 5] 0.00-10.01 sec 999 MBytes 838 Mbits/sec 0.015 ms 0/723751 (0%) receiver
[ 7] 0.00-10.00 sec 796 MBytes 668 Mbits/sec 0.000 ms 0/576448 (0%) sender
[ 7] 0.00-10.01 sec 796 MBytes 667 Mbits/sec 0.016 ms 0/576448 (0%) receiver
[ 9] 0.00-10.00 sec 689 MBytes 578 Mbits/sec 0.000 ms 0/498694 (0%) sender
[ 9] 0.00-10.01 sec 689 MBytes 577 Mbits/sec 0.013 ms 0/498694 (0%) receiver
[ 11] 0.00-10.00 sec 699 MBytes 587 Mbits/sec 0.000 ms 0/506458 (0%) sender
[ 11] 0.00-10.01 sec 699 MBytes 586 Mbits/sec 0.009 ms 0/506458 (0%) receiver
[ 13] 0.00-10.00 sec 882 MBytes 740 Mbits/sec 0.000 ms 0/638698 (0%) sender
[ 13] 0.00-10.01 sec 882 MBytes 739 Mbits/sec 0.006 ms 0/638698 (0%) receiver
[ 15] 0.00-10.00 sec 682 MBytes 572 Mbits/sec 0.000 ms 0/494045 (0%) sender
[ 15] 0.00-10.01 sec 682 MBytes 572 Mbits/sec 0.013 ms 0/494045 (0%) receiver
[ 17] 0.00-10.00 sec 980 MBytes 822 Mbits/sec 0.000 ms 0/709498 (0%) sender
.
.
(中略)
.
.
[119] 0.00-10.00 sec 926 MBytes 776 Mbits/sec 0.000 ms 0/670267 (0%) sender
[119] 0.00-10.01 sec 926 MBytes 776 Mbits/sec 0.011 ms 0/670267 (0%) receiver
[121] 0.00-10.00 sec 1.04 GBytes 897 Mbits/sec 0.000 ms 0/774703 (0%) sender
[121] 0.00-10.01 sec 1.04 GBytes 897 Mbits/sec 0.011 ms 0/774703 (0%) receiver
[123] 0.00-10.00 sec 885 MBytes 743 Mbits/sec 0.000 ms 0/641182 (0%) sender
[123] 0.00-10.01 sec 885 MBytes 742 Mbits/sec 0.010 ms 0/641182 (0%) receiver
[125] 0.00-10.00 sec 619 MBytes 519 Mbits/sec 0.000 ms 0/447922 (0%) sender
[125] 0.00-10.01 sec 619 MBytes 518 Mbits/sec 0.012 ms 0/447922 (0%) receiver
[127] 0.00-10.00 sec 699 MBytes 586 Mbits/sec 0.000 ms 0/506082 (0%) sender
[127] 0.00-10.01 sec 699 MBytes 586 Mbits/sec 0.011 ms 0/506082 (0%) receiver
[129] 0.00-10.00 sec 992 MBytes 832 Mbits/sec 0.000 ms 0/718578 (0%) sender
[129] 0.00-10.01 sec 992 MBytes 832 Mbits/sec 0.013 ms 0/718578 (0%) receiver
[131] 0.00-10.00 sec 694 MBytes 582 Mbits/sec 0.000 ms 0/502287 (0%) sender
[131] 0.00-10.01 sec 694 MBytes 581 Mbits/sec 0.016 ms 0/502287 (0%) receiver
[SUM] 0.00-10.00 sec 61.8 GBytes 53.1 Gbits/sec 0.000 ms 0/45830021 (0%) sender
[SUM] 0.00-10.01 sec 61.8 GBytes 53.0 Gbits/sec 0.012 ms 0/45830021 (0%) receiver
CPU Utilization: local/sender 4561.8% (94.6%u/4467.2%s), remote/receiver 1944.1% (158.7%u/1785.4%s)
iperf Done.
53.0Gbpsでした。
リージョンを跨いだ単一インスタンス間でもマルチフロートラフィックであれば高転送レートで転送できる
東京リージョンと大阪リージョンのEC2インスタンス間の転送速度を計測してみました。
リージョンを跨いだ単一インスタンス間でもマルチフロートラフィックであれば高転送レートで転送できることが分かりました。ただし、個人的にはリージョンを跨ぐ場合は「インスタンスで利用可能なネットワーク帯域幅の最大 50％ を利用できる」という制約を感じなかったのが気になります。
なお、本来であれば、測定時はセグメントサイズやら帯域幅、バッファ長、リングバッファなどを調整をしながら検証するのが望ましいとは思います。実際にインスタンス間で転送速度を改善したい場合はそういったパラメーターも変更してみると良いでしょう。
また、Transit Gatewayを間に挟む場合は注意が必要です。Transit Gatewayの最大帯域幅は50Gbpsです。場合によってはVPCピアリングを使ったバイパスを用意してあげた方が良いかもしれません。
帯域幅制限なし - Transit Gateway では、VPC 接続あたりの最大帯域幅 (バースト) は 50 Gbps です。VPC ピアリングには集約帯域幅がありません。個々のインスタンスのネットワークパフォーマンス制限とフロー制限 (プレイスメントグループ内では 10 Gbps、それ以外では 5 Gbps) が両方のオプションに適用されます。VPC ピアリングのみがプレイスメントグループをサポートしています。
Transit Gateway と VPC ピアリング - スケーラブルで安全なマルチ VPC の AWS ネットワークインフラストラクチャを構築する
この記事が誰かの助けになれば幸いです。
以上、AWS事業本部 コンサルティング部の のんピ(@non____97)でした!
