東京リージョンと大阪リージョンのEC2インスタンス間の転送速度を計測してみた

マルチフロートラフィックならEC2インスタンス間で70Gbpsは出せることが分かりました
2024.05.15

東京リージョンと大阪リージョンのEC2インスタンス間でどのぐらいの速度で転送できるのか気になる

こんにちは、のんピ(@non____97)です。

皆さんは東京リージョンと大阪リージョンのEC2インスタンス間でどのぐらいの転送速度が出るのか気になったことはありますか? 私はあります。

リージョン間の通信であってもvCPUが32個以上あるEC2インスタンスであればインスタンスで利用可能なネットワーク帯域幅の最大50%を利用できるため、それなりの転送速度が出そうです。

マルチフロートラフィック

インスタンスで使用できる集約マルチフロートラフィックの帯域幅は、トラフィックの宛先によって異なります。

リージョン内 – トラフィックは、インスタンスで使用可能な全ネットワーク帯域幅を利用することができます。

他のリージョンへの通信、インターネットゲートウェイ、Direct Connect、ローカルゲートウェイ (LGW) – トラフィックは、最低 32 個の vCPUs を搭載した現行世代のインスタンスで利用可能なネットワーク帯域幅の最大 50% を利用できます。32vCPUs未満の現世代のインスタンスの帯域幅は 5 Gbps に制限されています。

シングルフロートラフィック

インスタンスが同じクラスタープレイスメントグループにない場合、シングルフロートラフィックのベースライン帯域幅は 5 Gbps に制限されます。レイテンシーを減らし、シングルフロー帯域幅を増やすには、以下のいずれかをお試しください。

クラスタープレイスメントグループを使用すると、同じプレイスメントグループ内のインスタンスで最大 10 Gbps の帯域幅を実現できます。

または、任意の 2 つのエンドポイント間に複数のパスを設定することで、Multipath TCP (MPTCP) を使用して高帯域幅を実現できます。

同じサブネット内の対象インスタンスに ENA Express を設定して、それらのインスタンス間で最大 25 Gbps を実現します。

Amazon EC2 インスタンスのネットワーク帯域幅 - Amazon Elastic Compute Cloud

実際にどのぐらいの転送速度が出るのか試したことがなかったので、試してみました。

いきなりまとめ

  • リージョンを跨いだ単一インスタンス間でも、マルチフロートラフィックであれば高転送レートで転送できる
  • Transit Gateway経由の場合はバーストで50Gbpsまでしか出ないので注意
  • c6gn.mediumの場合 (単位はGbps)
    • c6gn.mediumのネットワーク帯域幅は最大16Gbps
プロトコル (バッファサイズ調整有無) / 多重度 1 4 8
TCP (バッファサイズ調整なし) 2.92 - -
TCP (バッファサイズ調整あり) 4.75 11.7 11.7
UDP (バッファサイズ調整なし) 2.19 - -
UDP (バッファサイズ調整あり) 2.56 3.39 3.59
  • c6gn.12xlargeの場合 (単位はGbps)
    • c6gn.12xlargeのネットワーク帯域幅は75Gbps
プロトコル (バッファサイズ調整有無) / 多重度 1 4 8 16 32 64 96
TCP (バッファサイズ調整なし) 2.95 12.2 24.5 47.4 66.4 71 71.2
TCP (バッファサイズ調整あり) 4.75 - - 56.7 69.8 71.1 -
UDP (バッファサイズ調整なし) 4.75 19.1 - - - - -
UDP (バッファサイズ調整あり) - - - 38.5 - 53 -

検証環境

検証環境は以下のとおりです。

東京リージョンと大阪リージョンのEC2インスタンス間で転送速度を確認してみた検証環境構成図

  • OS : Amazon Linux 2023 (al2023-ami-2023.4.20240429.0-kernel-6.1-arm64)
  • インスタンスタイプ : c6gn.medium (ネットワーク帯域幅最大16Gbps)

東京リージョンのEC2インスタンスから大阪リージョンのEC2インスタンスへiperf3を叩いて、その時の転送速度を確認します。iperf3の詳細は以下記事をご覧ください。

レイテンシーは8msで安定しています。

リージョン間のレイテンシー

EC2インスタンスの各種設定は以下のとおりです。

$ sudo cpupower frequency-info
analyzing CPU 0:
  no or unknown cpufreq driver is active on this CPU
  CPUs which run at the same hardware frequency: Not Available
  CPUs which need to have their frequency coordinated by software: Not Available
  maximum transition latency:  Cannot determine or is not supported.
Not Available
  available cpufreq governors: Not Available
  Unable to determine current policy
  current CPU frequency: Unable to call hardware
  current CPU frequency:  Unable to call to kernel

$ sudo sysctl -a | grep -e wmem -e rmem -e mtu -e default_qdis
net.core.default_qdisc = fq_codel
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.ipv4.ip_forward_use_pmtu = 0
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.tcp_mtu_probe_floor = 48
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_rmem = 4096        131072  6291456
net.ipv4.tcp_wmem = 4096        20480   4194304
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv6.conf.all.accept_ra_mtu = 1
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.default.accept_ra_mtu = 1
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.ens5.accept_ra_mtu = 1
net.ipv6.conf.ens5.mtu = 9001
net.ipv6.conf.lo.accept_ra_mtu = 1
net.ipv6.conf.lo.mtu = 65536
net.ipv6.route.mtu_expires = 600
vm.lowmem_reserve_ratio = 256   256     32      0       0

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 0a:49:24:4f:07:c5 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-07615f08d54aa652f
    altname device-number-0

$ ethtool -g ens5
Ring parameters for ens5:
Pre-set maximums:
RX:             2048
RX Mini:        n/a
RX Jumbo:       n/a
TX:             512
Current hardware settings:
RX:             1024
RX Mini:        n/a
RX Jumbo:       n/a
TX:             512

$ cat /sys/devices/system/cpu/smt/active
0

c6gn.medium の場合

TCP 多重度 1

まずはTCPで試します。

大阪リージョンのEC2インスタンスで待ち受けておきます。

$ iperf3 -s -V
iperf 3.16
Linux ip-10-0-0-7.ap-northeast-3.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------

東京側から接続します。

$ iperf3 -c 10.0.0.7 -V
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 07:57:58 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: jvx6y2phl7vnzjwr3u3zfozqwystnleoddkf
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 56460 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   320 MBytes  2.68 Gbits/sec    0   6.03 MBytes
[  5]   1.00-2.00   sec   351 MBytes  2.94 Gbits/sec    0   6.03 MBytes
[  5]   2.00-3.00   sec   351 MBytes  2.94 Gbits/sec    0   6.03 MBytes
[  5]   3.00-4.00   sec   348 MBytes  2.92 Gbits/sec    0   6.03 MBytes
[  5]   4.00-5.00   sec   351 MBytes  2.94 Gbits/sec    0   6.03 MBytes
[  5]   5.00-6.00   sec   352 MBytes  2.95 Gbits/sec    0   6.03 MBytes
[  5]   6.00-7.00   sec   352 MBytes  2.96 Gbits/sec    0   6.03 MBytes
[  5]   7.00-8.00   sec   351 MBytes  2.95 Gbits/sec    0   6.03 MBytes
[  5]   8.00-9.00   sec   353 MBytes  2.96 Gbits/sec    0   6.03 MBytes
[  5]   9.00-10.00  sec   352 MBytes  2.95 Gbits/sec    0   6.03 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.40 GBytes  2.92 Gbits/sec    0             sender
[  5]   0.00-10.01  sec  3.40 GBytes  2.92 Gbits/sec                  receiver
CPU Utilization: local/sender 10.6% (0.2%u/10.4%s), remote/receiver 20.3% (0.4%u/19.9%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

2.92GBでした。

大阪側でも同様の値を示しています。

$ iperf3 -s -V
iperf 3.16
Linux ip-10-0-0-7.ap-northeast-3.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Time: Tue, 14 May 2024 07:57:58 GMT
Accepted connection from 172.31.8.203, port 56450
      Cookie: jvx6y2phl7vnzjwr3u3zfozqwystnleoddkf
      TCP MSS: 0 (default)
[  5] local 10.0.0.7 port 5201 connected to 172.31.8.203 port 56460
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   317 MBytes  2.65 Gbits/sec
[  5]   1.00-2.00   sec   350 MBytes  2.94 Gbits/sec
[  5]   2.00-3.00   sec   352 MBytes  2.95 Gbits/sec
[  5]   3.00-4.00   sec   348 MBytes  2.92 Gbits/sec
[  5]   4.00-5.00   sec   350 MBytes  2.94 Gbits/sec
[  5]   5.00-6.00   sec   352 MBytes  2.95 Gbits/sec
[  5]   6.00-7.00   sec   352 MBytes  2.96 Gbits/sec
[  5]   7.00-8.00   sec   352 MBytes  2.95 Gbits/sec
[  5]   8.00-9.00   sec   352 MBytes  2.96 Gbits/sec
[  5]   9.00-10.00  sec   352 MBytes  2.96 Gbits/sec
[  5]  10.00-10.01  sec  2.88 MBytes  2.85 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate
[  5] (sender statistics not available)
[  5]   0.00-10.01  sec  3.40 GBytes  2.92 Gbits/sec                  receiver
rcv_tcp_congestion cubic

それなりに速いと言えるのではないでしょうか。

何回か繰り返しましたが、いずれも3Gbpsほどでした。

UDP 多重度 1

続いてUDPでも試します。

$ iperf3 -c 10.0.0.7 -V -u -b 16G
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:08:20 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: htq536yhyuxz7xl2ideoqrfkn3sz2pxvckfb
      Target Bitrate: 16000000000
[  5] local 172.31.8.203 port 48620 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   580 MBytes  4.87 Gbits/sec  420065
[  5]   1.00-2.00   sec   579 MBytes  4.86 Gbits/sec  419572
[  5]   2.00-3.00   sec   579 MBytes  4.86 Gbits/sec  419517
[  5]   3.00-4.00   sec   579 MBytes  4.86 Gbits/sec  419578
[  5]   4.00-5.00   sec   579 MBytes  4.86 Gbits/sec  419564
[  5]   5.00-6.00   sec   579 MBytes  4.86 Gbits/sec  419549
[  5]   6.00-7.00   sec   579 MBytes  4.86 Gbits/sec  419534
[  5]   7.00-8.00   sec   579 MBytes  4.86 Gbits/sec  419573
[  5]   8.00-9.00   sec   579 MBytes  4.86 Gbits/sec  419558
[  5]   9.00-10.00  sec   579 MBytes  4.85 Gbits/sec  419548
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec  0.000 ms  0/4196058 (0%)  sender
[  5]   0.00-10.01  sec  2.55 GBytes  2.19 Gbits/sec  0.002 ms  2301865/4196058 (55%)  receiver
CPU Utilization: local/sender 74.3% (5.5%u/68.8%s), remote/receiver 94.4% (10.9%u/83.5%s)

iperf Done.

2.19Gbpsですね。ロストした量がそれなりに多いですね。

大阪側も2.19Gbpsで記録されていました。

iperf 3.16
Linux ip-10-0-0-7.ap-northeast-3.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
-----------------------------------------------------------
Server listening on 5201 (test #4)
-----------------------------------------------------------
Time: Tue, 14 May 2024 08:01:08 GMT
Accepted connection from 172.31.8.203, port 53226
      Cookie: vrvvnw4nxsviukpsqqa72jbw6e44rv6odptl
      Target Bitrate: 16000000000
[  5] local 10.0.0.7 port 5201 connected to 172.31.8.203 port 50881
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec   268 MBytes  2.25 Gbits/sec  0.001 ms  221974/416088 (53%)
[  5]   1.00-2.00   sec   275 MBytes  2.30 Gbits/sec  0.001 ms  221285/420138 (53%)
[  5]   2.00-3.00   sec   275 MBytes  2.30 Gbits/sec  0.001 ms  218408/417247 (52%)
[  5]   3.00-4.00   sec   275 MBytes  2.31 Gbits/sec  0.001 ms  220354/419700 (53%)
[  5]   4.00-5.00   sec   240 MBytes  2.01 Gbits/sec  0.007 ms  245757/419428 (59%)
[  5]   5.00-6.00   sec   239 MBytes  2.00 Gbits/sec  0.002 ms  246635/419639 (59%)
[  5]   6.00-7.00   sec   274 MBytes  2.30 Gbits/sec  0.002 ms  221069/419825 (53%)
[  5]   7.00-8.00   sec   240 MBytes  2.01 Gbits/sec  0.001 ms  244830/418634 (58%)
[  5]   8.00-9.00   sec   264 MBytes  2.22 Gbits/sec  0.001 ms  228876/420104 (54%)
[  5]   9.00-10.00  sec   260 MBytes  2.18 Gbits/sec  0.001 ms  232263/420800 (55%)
[  5]  10.00-10.01  sec  3.59 MBytes  3.40 Gbits/sec  0.008 ms  1845/4446 (41%)
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5] (sender statistics not available)
[  5]   0.00-10.01  sec  2.55 GBytes  2.19 Gbits/sec  0.008 ms  2303296/4196049 (55%)  receiver
iperf 3.16
Linux ip-10-0-0-7.ap-northeast-3.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64

バッファサイズの変更をしたのち、UDP 多重度 1

大阪のEC2インスタンスを確認すると、バッファが溢れていそうです。

$ netstat -su
IcmpMsg:
    InType3: 3
    OutType3: 3
Udp:
    1894261 packets received
    3 packets to unknown port received
    2301503 packet receive errors
    72 packets sent
    2301503 receive buffer errors
    0 send buffer errors
UdpLite:
IpExt:
    InOctets: 6193293239
    OutOctets: 157453
    InNoECTPkts: 4197024
MPTcpExt:

大阪のEC2インスタンスのバッファサイズを変更してあげます。

$ sudo sysctl -w net.ipv4.tcp_wmem="65536 33554432"
$ sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 33554432"
$ sudo sysctl -w net.core.wmem_default=67108864
$ sudo sysctl -w net.core.rmem_default=67108864
$ sudo sysctl -w net.core.wmem_max=67108864
$ sudo sysctl -w net.core.rmem_max=67108864
$ sudo sysctl -a | grep -e wmem -e rmem
net.core.rmem_default = 2147483647
net.core.rmem_max = 2147483647
net.core.wmem_default = 2147483647
net.core.wmem_max = 2147483647
net.ipv4.tcp_rmem = 4096        67108864        1073741824
net.ipv4.tcp_wmem = 4096        67108864        1073741824
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
vm.lowmem_reserve_ratio = 256   256     32      0       0

再度、UDPを試してみます。

$ iperf3 -c 10.0.0.7 -V -u -b 16G
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:12:42 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: 55mtzbjqmr6fy46lxed24cjy7bb4skfujkqh
      Target Bitrate: 16000000000
[  5] local 172.31.8.203 port 40098 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   589 MBytes  4.94 Gbits/sec  426827
[  5]   1.00-2.00   sec   588 MBytes  4.93 Gbits/sec  425487
[  5]   2.00-3.00   sec   586 MBytes  4.92 Gbits/sec  424541
[  5]   3.00-4.00   sec   577 MBytes  4.84 Gbits/sec  417728
[  5]   4.00-5.00   sec   587 MBytes  4.93 Gbits/sec  425284
[  5]   5.00-6.00   sec   583 MBytes  4.89 Gbits/sec  422013
[  5]   6.00-7.00   sec   586 MBytes  4.91 Gbits/sec  424155
[  5]   7.00-8.00   sec   588 MBytes  4.93 Gbits/sec  425808
[  5]   8.00-9.00   sec   584 MBytes  4.90 Gbits/sec  422736
[  5]   9.00-10.00  sec   583 MBytes  4.89 Gbits/sec  422505
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  5.71 GBytes  4.91 Gbits/sec  0.000 ms  0/4237084 (0%)  sender
[  5]   0.00-10.01  sec  2.99 GBytes  2.56 Gbits/sec  0.002 ms  1917865/4133274 (46%)  receiver
CPU Utilization: local/sender 98.2% (7.7%u/90.5%s), remote/receiver 99.7% (11.0%u/88.7%s)

iperf Done.

あまり結果は変わりありませんね。

この後、送信側のバッファサイズや-b 4Gに変更したりしましたが、結果は変わりありませんでした。

せっかくバッファサイズを変更したのでTCPでも再度試します。

$ iperf3 -c 10.0.0.7 -V -Z
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:21:39 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: q7iqnyimaj5a2dmnz3u5o4xebfhypr47ntj5
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 36828 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   555 MBytes  4.65 Gbits/sec    0   7.33 MBytes
[  5]   1.00-2.00   sec   571 MBytes  4.79 Gbits/sec    0   7.33 MBytes
[  5]   2.00-3.00   sec   572 MBytes  4.80 Gbits/sec    0   7.33 MBytes
[  5]   3.00-4.00   sec   570 MBytes  4.78 Gbits/sec    0   7.33 MBytes
[  5]   4.00-5.00   sec   571 MBytes  4.79 Gbits/sec    0   7.33 MBytes
[  5]   5.00-6.00   sec   570 MBytes  4.78 Gbits/sec    0   7.33 MBytes
[  5]   6.00-7.00   sec   570 MBytes  4.78 Gbits/sec    0   7.33 MBytes
[  5]   7.00-8.00   sec   569 MBytes  4.77 Gbits/sec    0   7.33 MBytes
[  5]   8.00-9.00   sec   562 MBytes  4.71 Gbits/sec    0   7.33 MBytes
[  5]   9.00-10.00  sec   571 MBytes  4.79 Gbits/sec  135   5.19 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.55 GBytes  4.76 Gbits/sec  135             sender
[  5]   0.00-10.01  sec  5.53 GBytes  4.75 Gbits/sec                  receiver
CPU Utilization: local/sender 7.9% (0.1%u/7.8%s), remote/receiver 29.3% (0.9%u/28.4%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

4.75Gbpsと1Gbpsほど転送速度が速くなったように思えます。

TCP 多重度 4

そのまま多重度を増やしていった場合の転送速度の影響を確認しましょう。

まずは多重度4です。

$ iperf3 -c 10.0.0.7 -V -Z -P 4
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:22:45 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: nsxwunjybnhowfxn5roetw3scvxv2jqmqfpf
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 60518 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 60524 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 60534 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 60546 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 4 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   361 MBytes  3.02 Gbits/sec    0   7.94 MBytes
[  7]   0.00-1.00   sec   361 MBytes  3.03 Gbits/sec    0   8.68 MBytes
[  9]   0.00-1.00   sec   336 MBytes  2.82 Gbits/sec    0   7.18 MBytes
[ 11]   0.00-1.00   sec   363 MBytes  3.04 Gbits/sec    0   9.83 MBytes
[SUM]   0.00-1.00   sec  1.39 GBytes  11.9 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   354 MBytes  2.97 Gbits/sec    0   8.62 MBytes
[  7]   1.00-2.00   sec   369 MBytes  3.10 Gbits/sec    0   9.17 MBytes
[  9]   1.00-2.00   sec   356 MBytes  2.98 Gbits/sec    0   8.29 MBytes
[ 11]   1.00-2.00   sec   368 MBytes  3.08 Gbits/sec    0   9.83 MBytes
[SUM]   1.00-2.00   sec  1.41 GBytes  12.1 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   358 MBytes  3.00 Gbits/sec    0   8.62 MBytes
[  7]   2.00-3.00   sec   352 MBytes  2.95 Gbits/sec    0   9.63 MBytes
[  9]   2.00-3.00   sec   345 MBytes  2.89 Gbits/sec    0   8.76 MBytes
[ 11]   2.00-3.00   sec   352 MBytes  2.95 Gbits/sec    0   10.3 MBytes
[SUM]   2.00-3.00   sec  1.37 GBytes  11.8 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   308 MBytes  2.58 Gbits/sec    0   9.17 MBytes
[  7]   3.00-4.00   sec   329 MBytes  2.76 Gbits/sec    0   9.63 MBytes
[  9]   3.00-4.00   sec   310 MBytes  2.60 Gbits/sec    0   8.76 MBytes
[ 11]   3.00-4.00   sec   322 MBytes  2.70 Gbits/sec    0   10.3 MBytes
[SUM]   3.00-4.00   sec  1.24 GBytes  10.6 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   350 MBytes  2.94 Gbits/sec    0   9.17 MBytes
[  7]   4.00-5.00   sec   348 MBytes  2.92 Gbits/sec    0   9.63 MBytes
[  9]   4.00-5.00   sec   355 MBytes  2.98 Gbits/sec    0   9.26 MBytes
[ 11]   4.00-5.00   sec   360 MBytes  3.02 Gbits/sec    0   10.3 MBytes
[SUM]   4.00-5.00   sec  1.38 GBytes  11.9 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   354 MBytes  2.97 Gbits/sec    0   10.6 MBytes
[  7]   5.00-6.00   sec   367 MBytes  3.07 Gbits/sec    0   12.3 MBytes
[  9]   5.00-6.00   sec   352 MBytes  2.95 Gbits/sec    0   11.9 MBytes
[ 11]   5.00-6.00   sec   358 MBytes  3.00 Gbits/sec    0   12.7 MBytes
[SUM]   5.00-6.00   sec  1.40 GBytes  12.0 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   305 MBytes  2.56 Gbits/sec    0   12.5 MBytes
[  7]   6.00-7.00   sec   380 MBytes  3.19 Gbits/sec    0   14.0 MBytes
[  9]   6.00-7.00   sec   353 MBytes  2.96 Gbits/sec    0   13.2 MBytes
[ 11]   6.00-7.00   sec   346 MBytes  2.90 Gbits/sec    0   15.5 MBytes
[SUM]   6.00-7.00   sec  1.35 GBytes  11.6 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   346 MBytes  2.90 Gbits/sec    0   13.2 MBytes
[  7]   7.00-8.00   sec   359 MBytes  3.01 Gbits/sec    0   14.7 MBytes
[  9]   7.00-8.00   sec   342 MBytes  2.87 Gbits/sec    0   13.9 MBytes
[ 11]   7.00-8.00   sec   361 MBytes  3.03 Gbits/sec    0   16.2 MBytes
[SUM]   7.00-8.00   sec  1.38 GBytes  11.8 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   343 MBytes  2.88 Gbits/sec    0   13.2 MBytes
[  7]   8.00-9.00   sec   365 MBytes  3.06 Gbits/sec    0   14.7 MBytes
[  9]   8.00-9.00   sec   356 MBytes  2.99 Gbits/sec    0   13.9 MBytes
[ 11]   8.00-9.00   sec   367 MBytes  3.08 Gbits/sec    0   16.2 MBytes
[SUM]   8.00-9.00   sec  1.40 GBytes  12.0 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   343 MBytes  2.88 Gbits/sec    0   13.2 MBytes
[  7]   9.00-10.00  sec   355 MBytes  2.98 Gbits/sec    0   14.7 MBytes
[  9]   9.00-10.00  sec   353 MBytes  2.96 Gbits/sec    0   13.9 MBytes
[ 11]   9.00-10.00  sec   382 MBytes  3.20 Gbits/sec    0   16.2 MBytes
[SUM]   9.00-10.00  sec  1.40 GBytes  12.0 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.34 GBytes  2.87 Gbits/sec    0             sender
[  5]   0.00-10.02  sec  3.32 GBytes  2.84 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  3.50 GBytes  3.01 Gbits/sec    0             sender
[  7]   0.00-10.02  sec  3.49 GBytes  2.99 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  3.38 GBytes  2.90 Gbits/sec    0             sender
[  9]   0.00-10.02  sec  3.35 GBytes  2.87 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  3.49 GBytes  3.00 Gbits/sec    0             sender
[ 11]   0.00-10.02  sec  3.44 GBytes  2.95 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec    0             sender
[SUM]   0.00-10.02  sec  13.6 GBytes  11.7 Gbits/sec                  receiver
CPU Utilization: local/sender 15.8% (0.1%u/15.7%s), remote/receiver 93.8% (0.0%u/93.8%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

11.7Gbpsになりました。かなり速いですね。代わりに1接続あたりの転送速度は若干落ちています。

TCP 多重度 8

続いて多重度8です。

$ iperf3 -c 10.0.0.7 -V -Z -P 8
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:23:56 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: wkkwtwi6bpgrhfy3l7gnefeo4dba4z4sal4s
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 44774 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 44778 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 44786 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 44792 connected to 10.0.0.7 port 5201
[ 13] local 172.31.8.203 port 44802 connected to 10.0.0.7 port 5201
[ 15] local 172.31.8.203 port 44816 connected to 10.0.0.7 port 5201
[ 17] local 172.31.8.203 port 44822 connected to 10.0.0.7 port 5201
[ 19] local 172.31.8.203 port 44836 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 8 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.75 MBytes  48.2 Mbits/sec    0    372 KBytes
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   161 MBytes   135 Mbits/sec    0             sender
[  5]   0.00-10.05  sec   161 MBytes   134 Mbits/sec                  receiver
[  7]   0.00-10.00  sec  1.99 GBytes  1.71 Gbits/sec    4             sender
[  7]   0.00-10.05  sec  1.98 GBytes  1.69 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  1.53 GBytes  1.32 Gbits/sec    3             sender
[  9]   0.00-10.05  sec  1.53 GBytes  1.31 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  2.03 GBytes  1.74 Gbits/sec    0             sender
[ 11]   0.00-10.05  sec  1.99 GBytes  1.70 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  2.10 GBytes  1.80 Gbits/sec    2             sender
[ 13]   0.00-10.05  sec  2.06 GBytes  1.76 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.92 GBytes  1.65 Gbits/sec    0             sender
[ 15]   0.00-10.05  sec  1.92 GBytes  1.64 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  2.03 GBytes  1.74 Gbits/sec    3             sender
[ 17]   0.00-10.05  sec  2.02 GBytes  1.73 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  2.03 GBytes  1.74 Gbits/sec    0             sender
[ 19]   0.00-10.05  sec  1.98 GBytes  1.69 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  13.8 GBytes  11.8 Gbits/sec   12             sender
[SUM]   0.00-10.05  sec  13.6 GBytes  11.7 Gbits/sec                  receiver
CPU Utilization: local/sender 26.9% (0.2%u/26.8%s), remote/receiver 94.0% (0.4%u/93.7%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

11.7Gbpsです。多重度4の場合と変わりありませんね。頭打ちのような気がします。

とはいえ、リージョン間のマルチフロートラフィックの場合は帯域幅の最大50%の制限があるところ、11.7Gbps/16Gbpsとかなり検討していると思います。

UDP 多重度 4

UDPでも多重度を変更した場合を確認してみましょう。

$ iperf3 -c 10.0.0.7 -V -u -b 16G -Z -P 4
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:25:12 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: iwtouiaqw4i43rmvjqstwa2z2m72xqsubyln
      Target Bitrate: 16000000000
[  5] local 172.31.8.203 port 41174 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 45334 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 34556 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 35014 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 4 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   183 MBytes  1.53 Gbits/sec  132504
[  7]   0.00-1.00   sec   183 MBytes  1.53 Gbits/sec  132477
[  9]   0.00-1.00   sec   184 MBytes  1.54 Gbits/sec  133197
[ 11]   0.00-1.00   sec   183 MBytes  1.54 Gbits/sec  132804
[SUM]   0.00-1.00   sec   733 MBytes  6.15 Gbits/sec  530982
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   183 MBytes  1.53 Gbits/sec  132171
[  7]   1.00-2.00   sec   184 MBytes  1.54 Gbits/sec  133149
[  9]   1.00-2.00   sec   183 MBytes  1.54 Gbits/sec  132816
[ 11]   1.00-2.00   sec   184 MBytes  1.54 Gbits/sec  133005
[SUM]   1.00-2.00   sec   733 MBytes  6.15 Gbits/sec  531141
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   184 MBytes  1.54 Gbits/sec  133275
[  7]   2.00-3.00   sec   184 MBytes  1.55 Gbits/sec  133581
[  9]   2.00-3.00   sec   185 MBytes  1.55 Gbits/sec  134179
[ 11]   2.00-3.00   sec   185 MBytes  1.55 Gbits/sec  134139
[SUM]   2.00-3.00   sec   739 MBytes  6.20 Gbits/sec  535174
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   184 MBytes  1.54 Gbits/sec  133337
[  7]   3.00-4.00   sec   185 MBytes  1.55 Gbits/sec  133981
[  9]   3.00-4.00   sec   184 MBytes  1.55 Gbits/sec  133542
[ 11]   3.00-4.00   sec   184 MBytes  1.54 Gbits/sec  133257
[SUM]   3.00-4.00   sec   738 MBytes  6.19 Gbits/sec  534117
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   184 MBytes  1.54 Gbits/sec  133345
[  7]   4.00-5.00   sec   184 MBytes  1.55 Gbits/sec  133576
[  9]   4.00-5.00   sec   184 MBytes  1.55 Gbits/sec  133462
[ 11]   4.00-5.00   sec   185 MBytes  1.56 Gbits/sec  134256
[SUM]   4.00-5.00   sec   738 MBytes  6.19 Gbits/sec  534639
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   184 MBytes  1.54 Gbits/sec  132900
[  7]   5.00-6.00   sec   185 MBytes  1.55 Gbits/sec  133838
[  9]   5.00-6.00   sec   185 MBytes  1.55 Gbits/sec  134078
[ 11]   5.00-6.00   sec   185 MBytes  1.55 Gbits/sec  134015
[SUM]   5.00-6.00   sec   739 MBytes  6.20 Gbits/sec  534831
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   184 MBytes  1.54 Gbits/sec  133056
[  7]   6.00-7.00   sec   185 MBytes  1.55 Gbits/sec  134135
[  9]   6.00-7.00   sec   185 MBytes  1.55 Gbits/sec  134062
[ 11]   6.00-7.00   sec   184 MBytes  1.55 Gbits/sec  133413
[SUM]   6.00-7.00   sec   738 MBytes  6.19 Gbits/sec  534666
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   185 MBytes  1.55 Gbits/sec  133625
[  7]   7.00-8.00   sec   185 MBytes  1.55 Gbits/sec  134149
[  9]   7.00-8.00   sec   185 MBytes  1.55 Gbits/sec  133702
[ 11]   7.00-8.00   sec   185 MBytes  1.55 Gbits/sec  133907
[SUM]   7.00-8.00   sec   739 MBytes  6.20 Gbits/sec  535383
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   185 MBytes  1.55 Gbits/sec  133655
[  7]   8.00-9.00   sec   185 MBytes  1.55 Gbits/sec  133950
[  9]   8.00-9.00   sec   185 MBytes  1.55 Gbits/sec  134065
[ 11]   8.00-9.00   sec   185 MBytes  1.55 Gbits/sec  133903
[SUM]   8.00-9.00   sec   740 MBytes  6.20 Gbits/sec  535573
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   184 MBytes  1.54 Gbits/sec  133343
[  7]   9.00-10.00  sec   185 MBytes  1.55 Gbits/sec  134257
[  9]   9.00-10.00  sec   186 MBytes  1.56 Gbits/sec  134366
[ 11]   9.00-10.00  sec   185 MBytes  1.55 Gbits/sec  133973
[SUM]   9.00-10.00  sec   740 MBytes  6.21 Gbits/sec  535939
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.80 GBytes  1.54 Gbits/sec  0.000 ms  0/1331211 (0%)  sender
[  5]   0.00-10.03  sec  1.00 GBytes   857 Mbits/sec  0.002 ms  586310/1328129 (44%)  receiver
[  7]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec  0.000 ms  0/1337093 (0%)  sender
[  7]   0.00-10.03  sec  1.00 GBytes   858 Mbits/sec  0.001 ms  582453/1325133 (44%)  receiver
[  9]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec  0.000 ms  0/1337469 (0%)  sender
[  9]   0.00-10.03  sec  1005 MBytes   841 Mbits/sec  0.001 ms  579001/1306840 (44%)  receiver
[ 11]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec  0.000 ms  0/1336672 (0%)  sender
[ 11]   0.00-10.03  sec   995 MBytes   833 Mbits/sec  0.009 ms  577540/1298296 (44%)  receiver
[SUM]   0.00-10.00  sec  7.20 GBytes  6.19 Gbits/sec  0.000 ms  0/5342445 (0%)  sender
[SUM]   0.00-10.03  sec  3.96 GBytes  3.39 Gbits/sec  0.003 ms  2325304/5258398 (44%)  receiver
CPU Utilization: local/sender 99.3% (7.6%u/91.7%s), remote/receiver 99.4% (5.8%u/93.7%s)

iperf Done.

3.39Gbpsですね。TCPと比べると遅い印象があります。

UDP 多重度 8

続いて多重度8です。

$ iperf3 -c 10.0.0.7 -V -u -b 16G -Z -P 8
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:25:49 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: bbjiqbot63iscimxahiex2454ghxlf4iyib7
      Target Bitrate: 16000000000
[  5] local 172.31.8.203 port 46586 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 41800 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 40048 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 42235 connected to 10.0.0.7 port 5201
[ 13] local 172.31.8.203 port 47925 connected to 10.0.0.7 port 5201
[ 15] local 172.31.8.203 port 53274 connected to 10.0.0.7 port 5201
[ 17] local 172.31.8.203 port 53656 connected to 10.0.0.7 port 5201
[ 19] local 172.31.8.203 port 40819 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 8 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  92.4 MBytes   775 Mbits/sec  66928
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   922 MBytes   774 Mbits/sec  0.000 ms  0/668002 (0%)  sender
[  5]   0.00-10.02  sec   544 MBytes   456 Mbits/sec  0.002 ms  253911/648182 (39%)  receiver
[  7]   0.00-10.00  sec   902 MBytes   757 Mbits/sec  0.000 ms  0/653095 (0%)  sender
[  7]   0.00-10.02  sec   529 MBytes   442 Mbits/sec  0.559 ms  269385/652113 (41%)  receiver
[  9]   0.00-10.00  sec   923 MBytes   774 Mbits/sec  0.000 ms  0/668512 (0%)  sender
[  9]   0.00-10.02  sec   544 MBytes   455 Mbits/sec  0.001 ms  248285/642022 (39%)  receiver
[ 11]   0.00-10.00  sec   918 MBytes   770 Mbits/sec  0.000 ms  0/664574 (0%)  sender
[ 11]   0.00-10.02  sec   549 MBytes   460 Mbits/sec  0.544 ms  251138/648720 (39%)  receiver
[ 13]   0.00-10.00  sec   923 MBytes   775 Mbits/sec  0.000 ms  0/668716 (0%)  sender
[ 13]   0.00-10.02  sec   541 MBytes   453 Mbits/sec  0.001 ms  258260/650296 (40%)  receiver
[ 15]   0.00-10.00  sec   900 MBytes   755 Mbits/sec  0.000 ms  0/651934 (0%)  sender
[ 15]   0.00-10.02  sec   525 MBytes   440 Mbits/sec  0.001 ms  271654/651934 (42%)  receiver
[ 17]   0.00-10.00  sec   902 MBytes   757 Mbits/sec  0.000 ms  0/653520 (0%)  sender
[ 17]   0.00-10.02  sec   523 MBytes   438 Mbits/sec  0.001 ms  274466/653520 (42%)  receiver
[ 19]   0.00-10.00  sec   903 MBytes   757 Mbits/sec  0.000 ms  0/653723 (0%)  sender
[ 19]   0.00-10.02  sec   533 MBytes   447 Mbits/sec  0.145 ms  266580/652796 (41%)  receiver
[SUM]   0.00-10.00  sec  7.12 GBytes  6.12 Gbits/sec  0.000 ms  0/5282076 (0%)  sender
[SUM]   0.00-10.02  sec  4.19 GBytes  3.59 Gbits/sec  0.157 ms  2093679/5199583 (40%)  receiver
CPU Utilization: local/sender 99.1% (8.3%u/90.8%s), remote/receiver 98.9% (5.6%u/93.3%s)

iperf Done.

3.59Gbpsです。あまり変わりありませんね。

c6gn.16xlarge の場合

次にc6gn.16xlargeの場合を試してみましょう。

c6gn.16xlargeのネットワーク帯域幅は100Gbpsです。これは期待できますね。

c6gn.16xlargeに変更

いざ起動

としようとしたところ、大阪リージョン側でキャパシティ不足で立ち上がりませんでした。

Insufficient capacity.

c6gn.12xlarge の場合

インスタンスタイプの変更

c6gn.16xlargeが立ち上がらないのであれば、c6gn.12xlargeはどうでしょう。

c6gn.12xlargeのネットワーク帯域幅は75Gbpsです。

c6gn.12xlargeに変更

こちらは両リージョンとも問題なく立ち上がりました。

EC2インスタンスを停止->起動したのでバッファサイズは元に戻っていました。

$ sudo sysctl -a | grep -e wmem -e rmem -e mtu -e default_qdis
net.core.default_qdisc = fq_codel
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.ipv4.ip_forward_use_pmtu = 0
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.tcp_mtu_probe_floor = 48
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_rmem = 4096        131072  6291456
net.ipv4.tcp_wmem = 4096        20480   4194304
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv6.conf.all.accept_ra_mtu = 1
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.default.accept_ra_mtu = 1
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.ens5.accept_ra_mtu = 1
net.ipv6.conf.ens5.mtu = 9001
net.ipv6.conf.lo.accept_ra_mtu = 1
net.ipv6.conf.lo.mtu = 65536
net.ipv6.route.mtu_expires = 600
vm.lowmem_reserve_ratio = 256   256     32      0       0

$ ethtool -g ens5
Ring parameters for ens5:
Pre-set maximums:
RX:             8192
RX Mini:        n/a
RX Jumbo:       n/a
TX:             512
Current hardware settings:
RX:             1024
RX Mini:        n/a
RX Jumbo:       n/a
TX:             512

TCP 多重度 1

TCPの多重度1から試していきます。

$ iperf3 -c 10.0.0.7 -V -Z
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:31:50 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: hjg2vf7634jcqjse5i3ctbi3cdy3uttjdlu5
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 44292 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   316 MBytes  2.65 Gbits/sec   72   4.21 MBytes
[  5]   1.00-2.00   sec   356 MBytes  2.99 Gbits/sec    0   4.21 MBytes
[  5]   2.00-3.00   sec   355 MBytes  2.98 Gbits/sec    0   4.21 MBytes
[  5]   3.00-4.00   sec   357 MBytes  2.99 Gbits/sec    0   4.21 MBytes
[  5]   4.00-5.00   sec   357 MBytes  2.99 Gbits/sec    0   4.21 MBytes
[  5]   5.00-6.00   sec   357 MBytes  2.99 Gbits/sec    0   4.21 MBytes
[  5]   6.00-7.00   sec   355 MBytes  2.98 Gbits/sec    0   4.21 MBytes
[  5]   7.00-8.00   sec   354 MBytes  2.97 Gbits/sec    0   4.21 MBytes
[  5]   8.00-9.00   sec   357 MBytes  2.99 Gbits/sec    0   4.21 MBytes
[  5]   9.00-10.00  sec   351 MBytes  2.94 Gbits/sec   33   3.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.43 GBytes  2.95 Gbits/sec  105             sender
[  5]   0.00-10.01  sec  3.43 GBytes  2.95 Gbits/sec                  receiver
CPU Utilization: local/sender 9.1% (0.0%u/9.1%s), remote/receiver 18.7% (0.6%u/18.2%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

2.95 Gbpsです。インスタンスタイプ変更前とあまり変わりありませんね。

TCP 多重度 4

続いて多重度4です。

$ iperf3 -c 10.0.0.7 -V -Z -P 4
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:32:18 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: b774kyln7lkghtnrjzwzyi6scg4jmp4xylpg
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 60466 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 60468 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 60478 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 60482 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 4 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   345 MBytes  2.89 Gbits/sec    0   6.00 MBytes
[  7]   0.00-1.00   sec   377 MBytes  3.16 Gbits/sec    0   6.01 MBytes
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.56 GBytes  3.06 Gbits/sec    0             sender
[  5]   0.00-10.01  sec  3.56 GBytes  3.05 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  3.88 GBytes  3.33 Gbits/sec    0             sender
[  7]   0.00-10.01  sec  3.88 GBytes  3.33 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  3.31 GBytes  2.85 Gbits/sec    0             sender
[  9]   0.00-10.01  sec  3.31 GBytes  2.84 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  3.42 GBytes  2.94 Gbits/sec    0             sender
[ 11]   0.00-10.01  sec  3.42 GBytes  2.93 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  14.2 GBytes  12.2 Gbits/sec    0             sender
[SUM]   0.00-10.01  sec  14.2 GBytes  12.2 Gbits/sec                  receiver
CPU Utilization: local/sender 37.0% (0.2%u/36.8%s), remote/receiver 77.9% (2.8%u/75.1%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

12.2Gbpsです。こちらもインスタンスタイプ変更前とあまり変わりありませんね。

TCP 多重度 8

次に多重度8です。

$ iperf3 -c 10.0.0.7 -V -Z -P 8
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:32:46 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: my4yjwzwcuikfqciajalkm3ij5jlodiqbiw7
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 56626 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 56628 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 56632 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.78 GBytes  3.25 Gbits/sec    0             sender
[  5]   0.00-10.01  sec  3.78 GBytes  3.24 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  3.13 GBytes  2.69 Gbits/sec    0             sender
[  7]   0.00-10.01  sec  3.13 GBytes  2.69 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  3.54 GBytes  3.04 Gbits/sec    0             sender
[  9]   0.00-10.01  sec  3.54 GBytes  3.04 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  3.90 GBytes  3.35 Gbits/sec    0             sender
[ 11]   0.00-10.01  sec  3.90 GBytes  3.34 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  3.77 GBytes  3.24 Gbits/sec    0             sender
[ 13]   0.00-10.01  sec  3.77 GBytes  3.23 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  3.27 GBytes  2.81 Gbits/sec    0             sender
[ 15]   0.00-10.01  sec  3.27 GBytes  2.80 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  3.79 GBytes  3.25 Gbits/sec    0             sender
[ 17]   0.00-10.01  sec  3.79 GBytes  3.25 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  3.43 GBytes  2.95 Gbits/sec    0             sender
[ 19]   0.00-10.01  sec  3.43 GBytes  2.94 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  28.6 GBytes  24.6 Gbits/sec    0             sender
[SUM]   0.00-10.01  sec  28.6 GBytes  24.5 Gbits/sec                  receiver
CPU Utilization: local/sender 74.2% (0.3%u/73.9%s), remote/receiver 158.3% (4.1%u/154.3%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

24.5Gbpsです。壁を超えた感がありますね。

TCP 多重度 16

さらに増やしていきましょう。多重度16です。

$ iperf3 -c 10.0.0.7 -V -Z -P 16
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:33:12 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: fctf673ypwsjdmwlllj6jrdttdvf72nbqdt5
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 52718 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 52730 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.68 GBytes  3.16 Gbits/sec    0             sender
[  5]   0.00-10.01  sec  3.68 GBytes  3.16 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  2.61 GBytes  2.24 Gbits/sec    0             sender
[  7]   0.00-10.01  sec  2.61 GBytes  2.24 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  3.68 GBytes  3.16 Gbits/sec    0             sender
[  9]   0.00-10.01  sec  3.68 GBytes  3.16 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  3.47 GBytes  2.98 Gbits/sec    0             sender
[ 11]   0.00-10.01  sec  3.47 GBytes  2.98 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  3.85 GBytes  3.31 Gbits/sec    0             sender
[ 13]   0.00-10.01  sec  3.85 GBytes  3.30 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  3.93 GBytes  3.37 Gbits/sec  1413             sender
[ 15]   0.00-10.01  sec  3.93 GBytes  3.37 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  2.00 GBytes  1.72 Gbits/sec  623             sender
[ 17]   0.00-10.01  sec  2.00 GBytes  1.72 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  3.74 GBytes  3.21 Gbits/sec    0             sender
[ 19]   0.00-10.01  sec  3.73 GBytes  3.21 Gbits/sec                  receiver
[ 21]   0.00-10.00  sec  2.93 GBytes  2.52 Gbits/sec    0             sender
[ 21]   0.00-10.01  sec  2.93 GBytes  2.51 Gbits/sec                  receiver
[ 23]   0.00-10.00  sec  3.89 GBytes  3.34 Gbits/sec    0             sender
[ 23]   0.00-10.01  sec  3.89 GBytes  3.34 Gbits/sec                  receiver
[ 25]   0.00-10.00  sec  3.66 GBytes  3.14 Gbits/sec    0             sender
[ 25]   0.00-10.01  sec  3.66 GBytes  3.14 Gbits/sec                  receiver
[ 27]   0.00-10.00  sec  3.26 GBytes  2.80 Gbits/sec    0             sender
[ 27]   0.00-10.01  sec  3.26 GBytes  2.80 Gbits/sec                  receiver
[ 29]   0.00-10.00  sec  3.67 GBytes  3.16 Gbits/sec    0             sender
[ 29]   0.00-10.01  sec  3.67 GBytes  3.15 Gbits/sec                  receiver
[ 31]   0.00-10.00  sec  3.83 GBytes  3.29 Gbits/sec    0             sender
[ 31]   0.00-10.01  sec  3.83 GBytes  3.29 Gbits/sec                  receiver
[ 33]   0.00-10.00  sec  3.91 GBytes  3.36 Gbits/sec    0             sender
[ 33]   0.00-10.01  sec  3.91 GBytes  3.36 Gbits/sec                  receiver
[ 35]   0.00-10.00  sec  3.60 GBytes  3.09 Gbits/sec    0             sender
[ 35]   0.00-10.01  sec  3.60 GBytes  3.09 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  55.7 GBytes  47.8 Gbits/sec  2036             sender
[SUM]   0.00-10.01  sec  55.7 GBytes  47.8 Gbits/sec                  receiver
CPU Utilization: local/sender 139.7% (1.0%u/138.7%s), remote/receiver 322.5% (7.6%u/314.9%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

47.4Gbpsでした。1コネクションの転送速度が安定して3Gbps前後であることからまだまだ余裕はありそうです。

TCP 多重度 32

次に多重度32です。

$ iperf3 -c 10.0.0.7 -V -Z -P 32
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:33:54 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: 2mb67aufkomsevpraug5iue5dhkxdcws4l7i
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 58936 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 58942 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.15 GBytes  1.84 Gbits/sec    0             sender
[  5]   0.00-10.01  sec  2.15 GBytes  1.84 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.47 GBytes  1.26 Gbits/sec    0             sender
[  7]   0.00-10.01  sec  1.47 GBytes  1.26 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  2.15 GBytes  1.84 Gbits/sec   37             sender
[  9]   0.00-10.01  sec  2.15 GBytes  1.84 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  2.83 GBytes  2.43 Gbits/sec    0             sender
[ 11]   0.00-10.01  sec  2.83 GBytes  2.43 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  2.75 GBytes  2.36 Gbits/sec    0             sender
[ 13]   0.00-10.01  sec  2.75 GBytes  2.36 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  3.69 GBytes  3.17 Gbits/sec    0             sender
[ 15]   0.00-10.01  sec  3.69 GBytes  3.16 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  3.39 GBytes  2.91 Gbits/sec  114             sender
[ 17]   0.00-10.01  sec  3.39 GBytes  2.91 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  2.28 GBytes  1.96 Gbits/sec    0             sender
[ 19]   0.00-10.01  sec  2.28 GBytes  1.96 Gbits/sec                  receiver
[ 21]   0.00-10.00  sec  2.26 GBytes  1.94 Gbits/sec   39             sender
[ 21]   0.00-10.01  sec  2.26 GBytes  1.94 Gbits/sec                  receiver
[ 23]   0.00-10.00  sec  2.78 GBytes  2.39 Gbits/sec    0             sender
[ 23]   0.00-10.01  sec  2.78 GBytes  2.39 Gbits/sec                  receiver
[ 25]   0.00-10.00  sec  2.47 GBytes  2.13 Gbits/sec   38             sender
[ 25]   0.00-10.01  sec  2.47 GBytes  2.12 Gbits/sec                  receiver
[ 27]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec    0             sender
[ 27]   0.00-10.01  sec  1.61 GBytes  1.38 Gbits/sec                  receiver
[ 29]   0.00-10.00  sec  2.50 GBytes  2.15 Gbits/sec    0             sender
[ 29]   0.00-10.01  sec  2.50 GBytes  2.15 Gbits/sec                  receiver
[ 31]   0.00-10.00  sec  3.39 GBytes  2.91 Gbits/sec    0             sender
[ 31]   0.00-10.01  sec  3.39 GBytes  2.91 Gbits/sec                  receiver
[ 33]   0.00-10.00  sec  1018 MBytes   854 Mbits/sec    0             sender
[ 33]   0.00-10.01  sec  1016 MBytes   851 Mbits/sec                  receiver
[ 35]   0.00-10.00  sec  1.98 GBytes  1.70 Gbits/sec    0             sender
[ 35]   0.00-10.01  sec  1.98 GBytes  1.70 Gbits/sec                  receiver
[ 37]   0.00-10.00  sec  3.04 GBytes  2.61 Gbits/sec    0             sender
[ 37]   0.00-10.01  sec  3.04 GBytes  2.61 Gbits/sec                  receiver
[ 39]   0.00-10.00  sec  2.84 GBytes  2.44 Gbits/sec    0             sender
[ 39]   0.00-10.01  sec  2.84 GBytes  2.44 Gbits/sec                  receiver
[ 41]   0.00-10.00  sec  3.24 GBytes  2.78 Gbits/sec    0             sender
[ 41]   0.00-10.01  sec  3.24 GBytes  2.78 Gbits/sec                  receiver
[ 43]   0.00-10.00  sec  3.26 GBytes  2.80 Gbits/sec    0             sender
[ 43]   0.00-10.01  sec  3.26 GBytes  2.80 Gbits/sec                  receiver
[ 45]   0.00-10.00  sec  3.09 GBytes  2.65 Gbits/sec    0             sender
[ 45]   0.00-10.01  sec  3.09 GBytes  2.65 Gbits/sec                  receiver
[ 47]   0.00-10.00  sec  3.17 GBytes  2.72 Gbits/sec    0             sender
[ 47]   0.00-10.01  sec  3.17 GBytes  2.72 Gbits/sec                  receiver
[ 49]   0.00-10.00  sec  1.99 GBytes  1.71 Gbits/sec   59             sender
[ 49]   0.00-10.01  sec  1.99 GBytes  1.71 Gbits/sec                  receiver
[ 51]   0.00-10.00  sec  2.56 GBytes  2.20 Gbits/sec    0             sender
[ 51]   0.00-10.01  sec  2.56 GBytes  2.19 Gbits/sec                  receiver
[ 53]   0.00-10.00  sec  2.19 GBytes  1.88 Gbits/sec   41             sender
[ 53]   0.00-10.01  sec  2.19 GBytes  1.88 Gbits/sec                  receiver
[ 55]   0.00-10.00  sec  2.43 GBytes  2.09 Gbits/sec    0             sender
[ 55]   0.00-10.01  sec  2.43 GBytes  2.09 Gbits/sec                  receiver
[ 57]   0.00-10.00  sec  2.78 GBytes  2.39 Gbits/sec    0             sender
[ 57]   0.00-10.01  sec  2.78 GBytes  2.39 Gbits/sec                  receiver
[ 59]   0.00-10.00  sec  1.91 GBytes  1.64 Gbits/sec    0             sender
[ 59]   0.00-10.01  sec  1.91 GBytes  1.64 Gbits/sec                  receiver
[ 61]   0.00-10.00  sec  1.47 GBytes  1.26 Gbits/sec    0             sender
[ 61]   0.00-10.01  sec  1.47 GBytes  1.26 Gbits/sec                  receiver
[ 63]   0.00-10.00  sec  1.50 GBytes  1.29 Gbits/sec    0             sender
[ 63]   0.00-10.01  sec  1.50 GBytes  1.29 Gbits/sec                  receiver
[ 65]   0.00-10.00  sec  1.62 GBytes  1.39 Gbits/sec    0             sender
[ 65]   0.00-10.01  sec  1.62 GBytes  1.39 Gbits/sec                  receiver
[ 67]   0.00-10.00  sec  1.62 GBytes  1.39 Gbits/sec    0             sender
[ 67]   0.00-10.01  sec  1.62 GBytes  1.39 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  77.4 GBytes  66.5 Gbits/sec  328             sender
[SUM]   0.00-10.01  sec  77.4 GBytes  66.4 Gbits/sec                  receiver
CPU Utilization: local/sender 110.9% (0.8%u/110.1%s), remote/receiver 460.0% (10.9%u/449.1%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

66.4Gbpsでした。1コネクションあたりの転送速度が少し落ちてきましたね。

TCP 多重度 64

多重度64です。

$ iperf3 -c 10.0.0.7 -V -Z -P 64
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:34:25 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: nbaiokldyhhkrreqfbxpbdli4ubx6kr3e6s2
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 60776 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 60780 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 60782 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 60788 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.18 GBytes  1.01 Gbits/sec  128             sender
[  5]   0.00-10.01  sec  1.17 GBytes  1.01 Gbits/sec                  receiver
[  7]   0.00-10.01  sec  1022 MBytes   857 Mbits/sec  180             sender
[  7]   0.00-10.01  sec  1020 MBytes   854 Mbits/sec                  receiver
[  9]   0.00-10.01  sec   866 MBytes   726 Mbits/sec  190             sender
[  9]   0.00-10.01  sec   864 MBytes   724 Mbits/sec                  receiver
.
.
(中略)
.
.
[119]   0.00-10.01  sec   762 MBytes   638 Mbits/sec                  receiver
[121]   0.00-10.01  sec   920 MBytes   771 Mbits/sec  151             sender
[121]   0.00-10.01  sec   918 MBytes   769 Mbits/sec                  receiver
[123]   0.00-10.01  sec   706 MBytes   592 Mbits/sec    0             sender
[123]   0.00-10.01  sec   704 MBytes   590 Mbits/sec                  receiver
[125]   0.00-10.01  sec  1.51 GBytes  1.30 Gbits/sec    0             sender
[125]   0.00-10.01  sec  1.51 GBytes  1.29 Gbits/sec                  receiver
[127]   0.00-10.01  sec  2.06 GBytes  1.77 Gbits/sec    0             sender
[127]   0.00-10.01  sec  2.06 GBytes  1.77 Gbits/sec                  receiver
[129]   0.00-10.01  sec  1.58 GBytes  1.35 Gbits/sec   54             sender
[129]   0.00-10.01  sec  1.57 GBytes  1.35 Gbits/sec                  receiver
[131]   0.00-10.01  sec  1.04 GBytes   889 Mbits/sec   18             sender
[131]   0.00-10.01  sec  1.03 GBytes   886 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  82.9 GBytes  71.2 Gbits/sec  6654             sender
[SUM]   0.00-10.01  sec  82.8 GBytes  71.0 Gbits/sec                  receiver
CPU Utilization: local/sender 87.2% (0.6%u/86.6%s), remote/receiver 519.9% (11.9%u/508.1%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

71.0Gbpsでした。1コネクションあたりの転送速度からしてここらが限界でしょう。

異なるリージョンであるにも関わらず70Gbpsで転送できるのは素晴らしいですね。

また、再送されたTCPセグメントが6,654とかなり増えてきました。本来であれば帯域幅を-b 2Gなどある程度絞ってあげた方がいいかもしれないですね。

TCP 多重度 96

最後に多重度96です。

$ iperf3 -c 10.0.0.7 -V -Z -P 96
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:35:32 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: hnro4ufbbc6yvv5apahsgo64dsb3c45xzano
      TCP MSS: 1448 (default)
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   710 MBytes   595 Mbits/sec  129             sender
[  5]   0.00-10.01  sec   708 MBytes   593 Mbits/sec                  receiver
.
.
(中略)
.
.
[195]   0.00-10.01  sec  1.17 GBytes  1.01 Gbits/sec   90             sender
[195]   0.00-10.01  sec  1.17 GBytes  1.01 Gbits/sec                  receiver
[SUM]   0.00-10.01  sec  83.2 GBytes  71.4 Gbits/sec  13217             sender
[SUM]   0.00-10.01  sec  83.0 GBytes  71.2 Gbits/sec                  receiver
CPU Utilization: local/sender 81.6% (1.2%u/80.4%s), remote/receiver 540.9% (13.3%u/527.6%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

71.2Gbpsでした。多重度64Gbpsで頭打ちだったようですね。

UDP 多重度 1

UDPでも多重度を変えて試してみます。

$ iperf3 -c 10.0.0.7 -V -Z -u -b 16G
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:37:15 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: 5bboi4prm4x7fc3gg6nl6of2t4fbjbir64wp
      Target Bitrate: 16000000000
[  5] local 172.31.8.203 port 52388 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   580 MBytes  4.87 Gbits/sec  420136
[  5]   1.00-2.00   sec   579 MBytes  4.86 Gbits/sec  419551
[  5]   2.00-3.00   sec   579 MBytes  4.86 Gbits/sec  419543
[  5]   3.00-4.00   sec   579 MBytes  4.86 Gbits/sec  419566
[  5]   4.00-5.00   sec   579 MBytes  4.86 Gbits/sec  419553
[  5]   5.00-6.00   sec   579 MBytes  4.86 Gbits/sec  419559
[  5]   6.00-7.00   sec   579 MBytes  4.86 Gbits/sec  419544
[  5]   7.00-8.00   sec   579 MBytes  4.86 Gbits/sec  419548
[  5]   8.00-9.00   sec   579 MBytes  4.86 Gbits/sec  419572
[  5]   9.00-10.00  sec   579 MBytes  4.86 Gbits/sec  419563
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec  0.000 ms  0/4196135 (0%)  sender
[  5]   0.00-10.01  sec  5.54 GBytes  4.75 Gbits/sec  0.001 ms  87853/4195512 (2.1%)  receiver
CPU Utilization: local/sender 72.4% (6.3%u/66.0%s), remote/receiver 199.6% (20.0%u/179.6%s)

iperf Done.

4.75Gbpsでした。メモリサイズが増えたからかロストしたデータグラムも少なそうです。

UDP 多重度 4

次に多重度4です。

$ iperf3 -c 10.0.0.7 -V -Z -u -b 16G -P 4
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:37:53 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: ijqxkpaz6xvxkhtqy6idzlj5wt3nv6huafgt
      Target Bitrate: 16000000000
[  5] local 172.31.8.203 port 41792 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 33319 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 32915 connected to 10.0.0.7 port 5201
[ 11] local 172.31.8.203 port 41212 connected to 10.0.0.7 port 5201
Starting Test: protocol: UDP, 4 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   580 MBytes  4.87 Gbits/sec  420035
[  7]   0.00-1.00   sec   544 MBytes  4.57 Gbits/sec  394207
[  9]   0.00-1.00   sec   580 MBytes  4.86 Gbits/sec  420002
[ 11]   0.00-1.00   sec   580 MBytes  4.87 Gbits/sec  420047
[SUM]   0.00-1.00   sec  2.23 GBytes  19.2 Gbits/sec  1654291
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   579 MBytes  4.86 Gbits/sec  419568
[  7]   1.00-2.00   sec   549 MBytes  4.61 Gbits/sec  397590
[  9]   1.00-2.00   sec   579 MBytes  4.86 Gbits/sec  419531
[ 11]   1.00-2.00   sec   579 MBytes  4.86 Gbits/sec  419559
[SUM]   1.00-2.00   sec  2.23 GBytes  19.2 Gbits/sec  1656248
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   579 MBytes  4.86 Gbits/sec  419525
[  7]   2.00-3.00   sec   550 MBytes  4.61 Gbits/sec  398189
[  9]   2.00-3.00   sec   579 MBytes  4.86 Gbits/sec  419576
[ 11]   2.00-3.00   sec   579 MBytes  4.86 Gbits/sec  419555
[SUM]   2.00-3.00   sec  2.23 GBytes  19.2 Gbits/sec  1656845
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   579 MBytes  4.86 Gbits/sec  419574
[  7]   3.00-4.00   sec   547 MBytes  4.59 Gbits/sec  396229
[  9]   3.00-4.00   sec   579 MBytes  4.86 Gbits/sec  419555
[ 11]   3.00-4.00   sec   579 MBytes  4.86 Gbits/sec  419566
[SUM]   3.00-4.00   sec  2.23 GBytes  19.2 Gbits/sec  1654924
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   579 MBytes  4.86 Gbits/sec  419542
[  7]   4.00-5.00   sec   546 MBytes  4.58 Gbits/sec  395182
[  9]   4.00-5.00   sec   579 MBytes  4.86 Gbits/sec  419551
[ 11]   4.00-5.00   sec   579 MBytes  4.86 Gbits/sec  419544
[SUM]   4.00-5.00   sec  2.23 GBytes  19.2 Gbits/sec  1653819
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   579 MBytes  4.86 Gbits/sec  419548
[  7]   5.00-6.00   sec   542 MBytes  4.55 Gbits/sec  392605
[  9]   5.00-6.00   sec   579 MBytes  4.86 Gbits/sec  419579
[ 11]   5.00-6.00   sec   579 MBytes  4.86 Gbits/sec  419563
[SUM]   5.00-6.00   sec  2.23 GBytes  19.1 Gbits/sec  1651295
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   579 MBytes  4.86 Gbits/sec  419579
[  7]   6.00-7.00   sec   550 MBytes  4.61 Gbits/sec  398216
[  9]   6.00-7.00   sec   579 MBytes  4.86 Gbits/sec  419551
[ 11]   6.00-7.00   sec   579 MBytes  4.86 Gbits/sec  419561
[SUM]   6.00-7.00   sec  2.23 GBytes  19.2 Gbits/sec  1656907
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   579 MBytes  4.86 Gbits/sec  419550
[  7]   7.00-8.00   sec   546 MBytes  4.58 Gbits/sec  395661
[  9]   7.00-8.00   sec   579 MBytes  4.86 Gbits/sec  419549
[ 11]   7.00-8.00   sec   579 MBytes  4.86 Gbits/sec  419556
[SUM]   7.00-8.00   sec  2.23 GBytes  19.2 Gbits/sec  1654316
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   579 MBytes  4.86 Gbits/sec  419569
[  7]   8.00-9.00   sec   546 MBytes  4.58 Gbits/sec  395390
[  9]   8.00-9.00   sec   579 MBytes  4.86 Gbits/sec  419548
[ 11]   8.00-9.00   sec   579 MBytes  4.86 Gbits/sec  419545
[SUM]   8.00-9.00   sec  2.23 GBytes  19.2 Gbits/sec  1654052
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   579 MBytes  4.86 Gbits/sec  419532
[  7]   9.00-10.00  sec   542 MBytes  4.55 Gbits/sec  392531
[  9]   9.00-10.00  sec   579 MBytes  4.86 Gbits/sec  419527
[ 11]   9.00-10.00  sec   579 MBytes  4.86 Gbits/sec  419563
[SUM]   9.00-10.00  sec  2.23 GBytes  19.1 Gbits/sec  1651153
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec  0.000 ms  0/4196022 (0%)  sender
[  5]   0.00-10.01  sec  5.65 GBytes  4.85 Gbits/sec  0.001 ms  3557/4195779 (0.085%)  receiver
[  7]   0.00-10.00  sec  5.33 GBytes  4.58 Gbits/sec  0.000 ms  0/3955800 (0%)  sender
[  7]   0.00-10.01  sec  5.33 GBytes  4.58 Gbits/sec  0.001 ms  456/3955798 (0.012%)  receiver
[  9]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec  0.000 ms  0/4195969 (0%)  sender
[  9]   0.00-10.01  sec  5.66 GBytes  4.86 Gbits/sec  0.002 ms  0/4195969 (0%)  receiver
[ 11]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec  0.000 ms  0/4196059 (0%)  sender
[ 11]   0.00-10.01  sec  5.65 GBytes  4.85 Gbits/sec  0.001 ms  9431/4196059 (0.22%)  receiver
[SUM]   0.00-10.00  sec  22.3 GBytes  19.2 Gbits/sec  0.000 ms  0/16543850 (0%)  sender
[SUM]   0.00-10.01  sec  22.3 GBytes  19.1 Gbits/sec  0.001 ms  13444/16543605 (0.081%)  receiver
CPU Utilization: local/sender 324.9% (34.4%u/290.5%s), remote/receiver 439.4% (51.9%u/387.5%s)

iperf Done.

19.1Gbpsでした。TCPの多重度4と比較すると多少速いですね。

バッファサイズ変更

Fasterdata Knowledge BaseのLinux Tuningを参考に東京と大阪どちらのEC2インスタンスもバッファサイズを変更してみます。

$ sudo sysctl -w net.ipv4.tcp_wmem="4096 67108864 1073741824"
$ sudo sysctl -w net.ipv4.tcp_rmem="4096 67108864 1073741824"
$ sudo sysctl -w net.core.wmem_default=2147483647
$ sudo sysctl -w net.core.rmem_default=2147483647
$ sudo sysctl -w net.core.wmem_max=2147483647
$ sudo sysctl -w net.core.rmem_max=2147483647

$ sudo sysctl -a | grep -e wmem -e rmem
net.core.rmem_default = 2147483647
net.core.rmem_max = 2147483647
net.core.wmem_default = 2147483647
net.core.wmem_max = 2147483647
net.ipv4.tcp_rmem = 4096        67108864        1073741824
net.ipv4.tcp_wmem = 4096        67108864        1073741824
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
vm.lowmem_reserve_ratio = 256   256     32      0       0

TCP 多重度 1

再度TCPから確認していきます。

$ iperf3 -c 10.0.0.7 -V -Z
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:40:12 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: yho6tfdmzv4552akmdryyh2typziyvrzu6ow
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 57492 connected to 10.0.0.7 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   580 MBytes  4.86 Gbits/sec    0   6.85 MBytes
[  5]   1.00-2.00   sec   585 MBytes  4.91 Gbits/sec    0   6.85 MBytes
[  5]   2.00-3.00   sec   563 MBytes  4.72 Gbits/sec    0   6.85 MBytes
[  5]   3.00-4.00   sec   562 MBytes  4.72 Gbits/sec    0   6.85 MBytes
[  5]   4.00-5.00   sec   585 MBytes  4.91 Gbits/sec    0   6.85 MBytes
[  5]   5.00-6.00   sec   562 MBytes  4.72 Gbits/sec    0   6.85 MBytes
[  5]   6.00-7.00   sec   562 MBytes  4.72 Gbits/sec    0   6.85 MBytes
[  5]   7.00-8.00   sec   585 MBytes  4.90 Gbits/sec    0   6.85 MBytes
[  5]   8.00-9.00   sec   562 MBytes  4.72 Gbits/sec    0   6.85 MBytes
[  5]   9.00-10.00  sec   562 MBytes  4.72 Gbits/sec    0   6.85 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.58 GBytes  4.79 Gbits/sec    0             sender
[  5]   0.00-10.01  sec  5.54 GBytes  4.75 Gbits/sec                  receiver
CPU Utilization: local/sender 6.9% (0.1%u/6.8%s), remote/receiver 28.9% (1.0%u/27.9%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

4.75Gbpsとシングルフロートラフィックの上限5Gbps近く出ていますね。

TCP 多重度 16

順当に1コネクションあたりの転送速度が増していそうなので、多重度を一気に16に増やしてみます。

$ iperf3 -c 10.0.0.7 -V -Z -P 16
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:40:56 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: deopdqcynvkgo3yosp6zhhqxqmckrz3ani76
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 51372 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 51382 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 51386 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.62 GBytes  3.97 Gbits/sec  497             sender
[  5]   0.00-10.01  sec  4.58 GBytes  3.93 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  3.19 GBytes  2.74 Gbits/sec    0             sender
[  7]   0.00-10.01  sec  3.14 GBytes  2.70 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  4.44 GBytes  3.81 Gbits/sec  256             sender
[  9]   0.00-10.01  sec  4.40 GBytes  3.78 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  4.56 GBytes  3.91 Gbits/sec  478             sender
[ 11]   0.00-10.01  sec  4.52 GBytes  3.88 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  4.84 GBytes  4.16 Gbits/sec  226             sender
[ 13]   0.00-10.01  sec  4.79 GBytes  4.11 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  4.97 GBytes  4.27 Gbits/sec  268             sender
[ 15]   0.00-10.01  sec  4.92 GBytes  4.22 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  5.15 GBytes  4.43 Gbits/sec  258             sender
[ 17]   0.00-10.01  sec  5.10 GBytes  4.37 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  4.85 GBytes  4.17 Gbits/sec  180             sender
[ 19]   0.00-10.01  sec  4.81 GBytes  4.13 Gbits/sec                  receiver
[ 21]   0.00-10.00  sec  2.64 GBytes  2.26 Gbits/sec   61             sender
[ 21]   0.00-10.01  sec  2.59 GBytes  2.23 Gbits/sec                  receiver
[ 23]   0.00-10.00  sec  2.53 GBytes  2.18 Gbits/sec   75             sender
[ 23]   0.00-10.01  sec  2.48 GBytes  2.13 Gbits/sec                  receiver
[ 25]   0.00-10.00  sec  4.27 GBytes  3.67 Gbits/sec  170             sender
[ 25]   0.00-10.01  sec  4.22 GBytes  3.62 Gbits/sec                  receiver
[ 27]   0.00-10.00  sec  4.76 GBytes  4.08 Gbits/sec   52             sender
[ 27]   0.00-10.01  sec  4.70 GBytes  4.04 Gbits/sec                  receiver
[ 29]   0.00-10.00  sec  5.39 GBytes  4.63 Gbits/sec   46             sender
[ 29]   0.00-10.01  sec  5.36 GBytes  4.60 Gbits/sec                  receiver
[ 31]   0.00-10.00  sec  3.39 GBytes  2.91 Gbits/sec  341             sender
[ 31]   0.00-10.01  sec  3.35 GBytes  2.87 Gbits/sec                  receiver
[ 33]   0.00-10.00  sec  2.45 GBytes  2.10 Gbits/sec    0             sender
[ 33]   0.00-10.01  sec  2.40 GBytes  2.06 Gbits/sec                  receiver
[ 35]   0.00-10.00  sec  4.78 GBytes  4.11 Gbits/sec  504             sender
[ 35]   0.00-10.01  sec  4.74 GBytes  4.06 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  66.8 GBytes  57.4 Gbits/sec  3412             sender
[SUM]   0.00-10.01  sec  66.1 GBytes  56.7 Gbits/sec                  receiver
CPU Utilization: local/sender 73.7% (0.8%u/72.9%s), remote/receiver 334.1% (7.4%u/326.7%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

56.7Gbpsでした。バッファサイズを変更する前は47.4Gbpsだったので転送速度が改善されていそうです。

TCP 多重度 32

次に多重度32です。

$ iperf3 -c 10.0.0.7 -V -Z -P 32
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:42:08 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: 4mydnhhyspqsnx3pqqqbe5xdn42hw22ujvd6
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 41494 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 41504 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.48 GBytes  2.13 Gbits/sec   55             sender
[  5]   0.00-10.01  sec  2.43 GBytes  2.08 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  2.37 GBytes  2.03 Gbits/sec    0             sender
[  7]   0.00-10.01  sec  2.33 GBytes  2.00 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  3.43 GBytes  2.94 Gbits/sec   82             sender
[  9]   0.00-10.01  sec  3.39 GBytes  2.91 Gbits/sec                  receiver
.
.
(中略)
.
.
[ 59]   0.00-10.00  sec  1.97 GBytes  1.69 Gbits/sec   65             sender
[ 59]   0.00-10.01  sec  1.91 GBytes  1.64 Gbits/sec                  receiver
[ 61]   0.00-10.00  sec  1.95 GBytes  1.68 Gbits/sec   77             sender
[ 61]   0.00-10.01  sec  1.91 GBytes  1.64 Gbits/sec                  receiver
[ 63]   0.00-10.00  sec  2.75 GBytes  2.36 Gbits/sec  131             sender
[ 63]   0.00-10.01  sec  2.70 GBytes  2.32 Gbits/sec                  receiver
[ 65]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec    8             sender
[ 65]   0.00-10.01  sec  2.15 GBytes  1.85 Gbits/sec                  receiver
[ 67]   0.00-10.00  sec  2.19 GBytes  1.88 Gbits/sec    3             sender
[ 67]   0.00-10.01  sec  2.15 GBytes  1.84 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  82.9 GBytes  71.2 Gbits/sec  2501             sender
[SUM]   0.00-10.01  sec  81.4 GBytes  69.8 Gbits/sec                  receiver
CPU Utilization: local/sender 87.8% (0.8%u/87.0%s), remote/receiver 422.3% (12.5%u/409.8%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

69.8Gbpsでした。

TCP 多重度 64

続いて多重度64です。

$ iperf3 -c 10.0.0.7 -V -Z -P 64
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Time: Tue, 14 May 2024 08:42:47 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: q2rmrlwcrb46ikv6fpo5f4zlniitn37lvr5y
      TCP MSS: 1448 (default)
[  5] local 172.31.8.203 port 48948 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 48956 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.06 GBytes   909 Mbits/sec   64             sender
[  5]   0.00-10.02  sec  1.00 GBytes   858 Mbits/sec                  receiver
[  7]   0.00-10.01  sec   995 MBytes   834 Mbits/sec   12             sender
[  7]   0.00-10.02  sec   952 MBytes   798 Mbits/sec                  receiver
[  9]   0.00-10.01  sec  2.07 GBytes  1.78 Gbits/sec   52             sender
[  9]   0.00-10.02  sec  2.02 GBytes  1.74 Gbits/sec                  receiver
[ 11]   0.00-10.01  sec  1.14 GBytes   975 Mbits/sec  230             sender
[ 11]   0.00-10.02  sec  1.09 GBytes   933 Mbits/sec                  receiver
[ 13]   0.00-10.01  sec  1.20 GBytes  1.03 Gbits/sec  140             sender
[ 13]   0.00-10.02  sec  1.15 GBytes   983 Mbits/sec                  receiver
.
.
(中略)
.
.
[123]   0.00-10.01  sec   779 MBytes   653 Mbits/sec  112             sender
[123]   0.00-10.02  sec   726 MBytes   608 Mbits/sec                  receiver
[125]   0.00-10.01  sec  1.26 GBytes  1.09 Gbits/sec  194             sender
[125]   0.00-10.02  sec  1.21 GBytes  1.04 Gbits/sec                  receiver
[127]   0.00-10.01  sec  1.77 GBytes  1.52 Gbits/sec   24             sender
[127]   0.00-10.02  sec  1.72 GBytes  1.47 Gbits/sec                  receiver
[129]   0.00-10.01  sec   886 MBytes   743 Mbits/sec   32             sender
[129]   0.00-10.02  sec   830 MBytes   695 Mbits/sec                  receiver
[131]   0.00-10.01  sec   971 MBytes   814 Mbits/sec   15             sender
[131]   0.00-10.02  sec   918 MBytes   769 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  86.1 GBytes  73.9 Gbits/sec  7104             sender
[SUM]   0.00-10.02  sec  83.0 GBytes  71.1 Gbits/sec                  receiver
CPU Utilization: local/sender 90.0% (1.1%u/88.9%s), remote/receiver 496.4% (14.3%u/482.1%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

71.1Gbpsでした。バッファサイズ変更前とあまり変わりありませんね。

UDP 多重度 16

TCPでインスタンス上限の75Gbpsに近い値を出せたので満足ですが、UDPでも何回か試してみます。

まず、多重度16です。

$ iperf3 -c 10.0.0.7 -V -Z -u -b 16G -P 16
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:43:51 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: 22yrdubjcyytsaw6fonholveldzfke5nz3ns
      Target Bitrate: 16000000000
[  5] local 172.31.8.203 port 39826 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 55985 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 35873 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  4.30 GBytes  3.70 Gbits/sec  0.000 ms  0/3191957 (0%)  sender
[  5]   0.00-10.01  sec  4.30 GBytes  3.69 Gbits/sec  0.002 ms  0/3191957 (0%)  receiver
[  7]   0.00-10.00  sec  3.11 GBytes  2.68 Gbits/sec  0.000 ms  0/2309671 (0%)  sender
[  7]   0.00-10.01  sec  1.48 GBytes  1.27 Gbits/sec  0.005 ms  1206485/2303519 (52%)  receiver
[  9]   0.00-10.00  sec  3.69 GBytes  3.17 Gbits/sec  0.000 ms  0/2738909 (0%)  sender
[  9]   0.00-10.01  sec  1.59 GBytes  1.36 Gbits/sec  0.007 ms  1555112/2733844 (57%)  receiver
[ 11]   0.00-10.00  sec  4.07 GBytes  3.49 Gbits/sec  0.000 ms  0/3015177 (0%)  sender
[ 11]   0.00-10.01  sec  1.36 GBytes  1.17 Gbits/sec  0.007 ms  1994688/3003591 (66%)  receiver
[ 13]   0.00-10.00  sec  3.46 GBytes  2.97 Gbits/sec  0.000 ms  0/2564061 (0%)  sender
[ 13]   0.00-10.01  sec  1.62 GBytes  1.39 Gbits/sec  0.007 ms  1358020/2559007 (53%)  receiver
[ 15]   0.00-10.00  sec  4.49 GBytes  3.86 Gbits/sec  0.000 ms  0/3332373 (0%)  sender
[ 15]   0.00-10.01  sec  4.49 GBytes  3.86 Gbits/sec  0.003 ms  137/3332373 (0.0041%)  receiver
[ 17]   0.00-10.00  sec  4.16 GBytes  3.57 Gbits/sec  0.000 ms  0/3082430 (0%)  sender
[ 17]   0.00-10.01  sec  1.83 GBytes  1.57 Gbits/sec  0.002 ms  1712705/3071249 (56%)  receiver
[ 19]   0.00-10.00  sec  3.33 GBytes  2.86 Gbits/sec  0.000 ms  0/2470818 (0%)  sender
[ 19]   0.00-10.01  sec  2.64 GBytes  2.26 Gbits/sec  0.003 ms  507865/2464426 (21%)  receiver
[ 21]   0.00-10.00  sec  4.07 GBytes  3.50 Gbits/sec  0.000 ms  0/3017352 (0%)  sender
[ 21]   0.00-10.01  sec  1.38 GBytes  1.18 Gbits/sec  0.012 ms  1984736/3005727 (66%)  receiver
[ 23]   0.00-10.00  sec  4.60 GBytes  3.95 Gbits/sec  0.000 ms  0/3410073 (0%)  sender
[ 23]   0.00-10.01  sec  4.60 GBytes  3.95 Gbits/sec  0.002 ms  0/3410073 (0%)  receiver
[ 25]   0.00-10.00  sec  4.06 GBytes  3.49 Gbits/sec  0.000 ms  0/3009308 (0%)  sender
[ 25]   0.00-10.01  sec  4.06 GBytes  3.48 Gbits/sec  0.002 ms  0/3009308 (0%)  receiver
[ 27]   0.00-10.00  sec  4.29 GBytes  3.69 Gbits/sec  0.000 ms  0/3182991 (0%)  sender
[ 27]   0.00-10.01  sec  3.02 GBytes  2.59 Gbits/sec  0.002 ms  933114/3171473 (29%)  receiver
[ 29]   0.00-10.00  sec  4.47 GBytes  3.84 Gbits/sec  0.000 ms  0/3310993 (0%)  sender
[ 29]   0.00-10.01  sec  4.47 GBytes  3.83 Gbits/sec  0.003 ms  0/3310993 (0%)  receiver
[ 31]   0.00-10.00  sec  3.62 GBytes  3.11 Gbits/sec  0.000 ms  0/2687322 (0%)  sender
[ 31]   0.00-10.01  sec  3.62 GBytes  3.11 Gbits/sec  0.003 ms  0/2687294 (0%)  receiver
[ 33]   0.00-10.00  sec  4.03 GBytes  3.46 Gbits/sec  0.000 ms  0/2991237 (0%)  sender
[ 33]   0.00-10.01  sec  2.91 GBytes  2.50 Gbits/sec  0.005 ms  827909/2986253 (28%)  receiver
[ 35]   0.00-10.00  sec  3.84 GBytes  3.29 Gbits/sec  0.000 ms  0/2844036 (0%)  sender
[ 35]   0.00-10.01  sec  1.44 GBytes  1.24 Gbits/sec  0.015 ms  1768709/2837588 (62%)  receiver
[SUM]   0.00-10.00  sec  63.6 GBytes  54.6 Gbits/sec  0.000 ms  0/47158708 (0%)  sender
[SUM]   0.00-10.01  sec  44.8 GBytes  38.5 Gbits/sec  0.005 ms  13849480/47078675 (29%)  receiver
CPU Utilization: local/sender 1579.0% (112.9%u/1466.1%s), remote/receiver 828.9% (95.1%u/733.8%s)

iperf Done.

38.5Gbpsでした。

UDP 多重度 64

最後に多重度64です。

$ iperf3 -c 10.0.0.7 -V -Z -u -b 16G -P 64
iperf 3.16
Linux ip-172-31-8-203.ap-northeast-1.compute.internal 6.1.87-99.174.amzn2023.aarch64 #1 SMP Tue Apr 23 00:06:47 UTC 2024 aarch64
Control connection MSS 1448
Setting UDP block size to 1448
Time: Tue, 14 May 2024 08:44:37 GMT
Connecting to host 10.0.0.7, port 5201
      Cookie: ocekrkbm2w6puttizmq3rzlrxbtrjs4y7f6v
      Target Bitrate: 16000000000
[  5] local 172.31.8.203 port 52683 connected to 10.0.0.7 port 5201
[  7] local 172.31.8.203 port 43005 connected to 10.0.0.7 port 5201
[  9] local 172.31.8.203 port 47483 connected to 10.0.0.7 port 5201
.
.
(中略)
.
.
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   999 MBytes   838 Mbits/sec  0.000 ms  0/723751 (0%)  sender
[  5]   0.00-10.01  sec   999 MBytes   838 Mbits/sec  0.015 ms  0/723751 (0%)  receiver
[  7]   0.00-10.00  sec   796 MBytes   668 Mbits/sec  0.000 ms  0/576448 (0%)  sender
[  7]   0.00-10.01  sec   796 MBytes   667 Mbits/sec  0.016 ms  0/576448 (0%)  receiver
[  9]   0.00-10.00  sec   689 MBytes   578 Mbits/sec  0.000 ms  0/498694 (0%)  sender
[  9]   0.00-10.01  sec   689 MBytes   577 Mbits/sec  0.013 ms  0/498694 (0%)  receiver
[ 11]   0.00-10.00  sec   699 MBytes   587 Mbits/sec  0.000 ms  0/506458 (0%)  sender
[ 11]   0.00-10.01  sec   699 MBytes   586 Mbits/sec  0.009 ms  0/506458 (0%)  receiver
[ 13]   0.00-10.00  sec   882 MBytes   740 Mbits/sec  0.000 ms  0/638698 (0%)  sender
[ 13]   0.00-10.01  sec   882 MBytes   739 Mbits/sec  0.006 ms  0/638698 (0%)  receiver
[ 15]   0.00-10.00  sec   682 MBytes   572 Mbits/sec  0.000 ms  0/494045 (0%)  sender
[ 15]   0.00-10.01  sec   682 MBytes   572 Mbits/sec  0.013 ms  0/494045 (0%)  receiver
[ 17]   0.00-10.00  sec   980 MBytes   822 Mbits/sec  0.000 ms  0/709498 (0%)  sender
.
.
(中略)
.
.
[119]   0.00-10.00  sec   926 MBytes   776 Mbits/sec  0.000 ms  0/670267 (0%)  sender
[119]   0.00-10.01  sec   926 MBytes   776 Mbits/sec  0.011 ms  0/670267 (0%)  receiver
[121]   0.00-10.00  sec  1.04 GBytes   897 Mbits/sec  0.000 ms  0/774703 (0%)  sender
[121]   0.00-10.01  sec  1.04 GBytes   897 Mbits/sec  0.011 ms  0/774703 (0%)  receiver
[123]   0.00-10.00  sec   885 MBytes   743 Mbits/sec  0.000 ms  0/641182 (0%)  sender
[123]   0.00-10.01  sec   885 MBytes   742 Mbits/sec  0.010 ms  0/641182 (0%)  receiver
[125]   0.00-10.00  sec   619 MBytes   519 Mbits/sec  0.000 ms  0/447922 (0%)  sender
[125]   0.00-10.01  sec   619 MBytes   518 Mbits/sec  0.012 ms  0/447922 (0%)  receiver
[127]   0.00-10.00  sec   699 MBytes   586 Mbits/sec  0.000 ms  0/506082 (0%)  sender
[127]   0.00-10.01  sec   699 MBytes   586 Mbits/sec  0.011 ms  0/506082 (0%)  receiver
[129]   0.00-10.00  sec   992 MBytes   832 Mbits/sec  0.000 ms  0/718578 (0%)  sender
[129]   0.00-10.01  sec   992 MBytes   832 Mbits/sec  0.013 ms  0/718578 (0%)  receiver
[131]   0.00-10.00  sec   694 MBytes   582 Mbits/sec  0.000 ms  0/502287 (0%)  sender
[131]   0.00-10.01  sec   694 MBytes   581 Mbits/sec  0.016 ms  0/502287 (0%)  receiver
[SUM]   0.00-10.00  sec  61.8 GBytes  53.1 Gbits/sec  0.000 ms  0/45830021 (0%)  sender
[SUM]   0.00-10.01  sec  61.8 GBytes  53.0 Gbits/sec  0.012 ms  0/45830021 (0%)  receiver
CPU Utilization: local/sender 4561.8% (94.6%u/4467.2%s), remote/receiver 1944.1% (158.7%u/1785.4%s)

iperf Done.

53.0Gbpsでした。

リージョンを跨いだ単一インスタンス間でもマルチフロートラフィックであれば高転送レートで転送できる

東京リージョンと大阪リージョンのEC2インスタンス間の転送速度を計測してみました。

リージョンを跨いだ単一インスタンス間でもマルチフロートラフィックであれば高転送レートで転送できることが分かりました。ただし、個人的にはリージョンを跨ぐ場合は「インスタンスで利用可能なネットワーク帯域幅の最大 50% を利用できる」という制約を感じなかったのが気になります。

なお、本来であれば、測定時はセグメントサイズやら帯域幅、バッファ長、リングバッファなどを調整をしながら検証するのが望ましいとは思います。実際にインスタンス間で転送速度を改善したい場合はそういったパラメーターも変更してみると良いでしょう。

また、Transit Gatewayを間に挟む場合は注意が必要です。Transit Gatewayの最大帯域幅は50Gbpsです。場合によってはVPCピアリングを使ったバイパスを用意してあげた方が良いかもしれません。

帯域幅制限なし - Transit Gateway では、VPC 接続あたりの最大帯域幅 (バースト) は 50 Gbps です。VPC ピアリングには集約帯域幅がありません。個々のインスタンスのネットワークパフォーマンス制限とフロー制限 (プレイスメントグループ内では 10 Gbps、それ以外では 5 Gbps) が両方のオプションに適用されます。VPC ピアリングのみがプレイスメントグループをサポートしています。

Transit Gateway と VPC ピアリング - スケーラブルで安全なマルチ VPC の AWS ネットワークインフラストラクチャを構築する

この記事が誰かの助けになれば幸いです。

以上、AWS事業本部 コンサルティング部の のんピ(@non____97)でした!