![[Amazon FSx for NetApp ONTAP] FlexGroupのリバランシング機能を試してみた](https://devio2023-media.developers.io/wp-content/uploads/2023/08/amazon-fsx-for-netapp-ontap.png)
[Amazon FSx for NetApp ONTAP] FlexGroupのリバランシング機能を試してみた
FlexGroupのコンスティチュエント間のデータ使用量の偏りが原因でパフォーマンスに影響が出ている時に
2024.04.08FlexGroupのコンスティチュエント間のデータ使用量の偏りが気になる
こんにちは、のんピ(@non____97)です。
皆さんはFlexGroupのコンスティチュエント間のデータ使用量の偏りが気になったことはありますか? 私はあります。
FlexGroupの内部では複数のFlexVolumeのコンスティチュエント(メンバーボリューム)を束ねて、1つのボリュームとして見せています。FlexGroupの詳細は以下記事をご覧ください。
FlexGroupを運用しているとのコンスティチュエント間のデータ使用量の偏りが出てくることがあります。特にコンスティチュエントを追加した場合は偏ります。ONTAP側で良きようにバランスを取って書き込みをしてくれますが、大きいファイルの追加や削除をした場合は不均衡になりがちです。
- ONTAP は常にメンバーボリュームの運用状態を監視し、 FlexGroup ボリュームの現在の状態に基づいて配置の決定を調整します。
- ONTAP 9.8 よりも前のバージョンでは、 1 つのメンバーボリュームが他のメンバーボリュームとのバランスが少し取れていない場合、取り込みの調整が行われない可能性があります。
- しかし、このメンバーボリュームが容量の 90% に近づくか、 512GB を超える容量不一致が発生する場合、 ONTAP は新しいデータをより積極的に配置して不均衡を解消します。
- この調整によって、他のメンバーボリュームへのリモートファイル配置が増えることになり、 FlexGroup ボリュームのパフォーマンスが低下する（ただし顕著ではない場合もある - 約 5~10% ）ことがあります。
コンスティチュエントのデータが偏るとCPU使用率が高まることがあるようです。
今までは時の流れに身を委ねるしかありませんでしたが、ONTAP 9.12.1以降では、コンスティチュエント間でファイルを無停止で移動することができるようになりました。
実際に試してみました。
いきなりまとめ
- 無停止でFlexGroupのコンスティチュエント間のファイルを移動できる
- リバランシング可能なファイルはSnapshotに含まれないものだけなど、注意点はいくつかある
- 詳細はこちらを参照
- スケジュール実行することも可能
- 定期実行ではなく、指定したタイミングに一回だけ実行
- 運用の中で頻繁に実行するものではなく、必要性に応じて実行するもの
やってみた
FlexGroupの作成
まず、FlexGroupボリュームを作成します。コンスティチュエントは2です。
::*> volume create -vserver svm -volume fg -aggr-list aggr1 -aggr-list-multiplier 2 -size 200GB
Notice: The FlexGroup volume "fg" will be created with the following number of constituents of size 100GB: 2.
Do you want to continue? {y|n}: y
Warning: You are attempting to create a FlexGroup volume in Vserver "svm", which currently uses 32-bit NFSv3 FSIDs and file IDs. This could result in collisions between different file IDs on the FlexGroup
volume. Modify the Vserver to use 64-bit NFSv3 identifiers before using the FlexGroup volume with the commands "set advanced; vserver nfs modify -vserver svm -v3-64bit-identifiers enabled".
Do you want to continue? {y|n}: y
[Job 58] Job succeeded: Successful
::*> volume show -volume fg*
Vserver Volume Aggregate State Type Size Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
svm fg - online RW 200GB 189.9GB 0%
svm fg__0001 aggr1 online RW 100GB 94.94GB 0%
svm fg__0002 aggr1 online RW 100GB 94.94GB 0%
3 entries were displayed.
::*> volume mount -volume fg -junction-path /fg
Queued private job: 29
テストファイルの作成
FlexGroupボリューム上にテストファイルを作成します。
$ sudo mkdir -p /mnt/fsxn/fg
$ sudo mount -t nfs svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg /mnt/fsxn/fg
$ df -hT -t nfs4
Filesystem Type Size Used Avail Use% Mounted on
svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg nfs4 190G 973M 190G 1% /mnt/fsxn/fg
$ for i in {33..48}; do
sudo dd if=/dev/urandom of=/mnt/fsxn/fg/random_pattern_binary_block_${i} bs=1M count=512
done
$ ls -l /mnt/fsxn/fg
total 8421888
-rw-r--r--. 1 root root 268435456 Apr 7 06:59 random_pattern_binary_block_1
-rw-r--r--. 1 root root 268435456 Apr 7 06:59 random_pattern_binary_block_10
-rw-r--r--. 1 root root 268435456 Apr 7 07:00 random_pattern_binary_block_11
.
.
(中略)
.
.
-rw-r--r--. 1 root root 268435456 Apr 7 06:59 random_pattern_binary_block_7
-rw-r--r--. 1 root root 268435456 Apr 7 06:59 random_pattern_binary_block_8
-rw-r--r--. 1 root root 268435456 Apr 7 06:59 random_pattern_binary_block_9
$ df -hT -t nfs4
Filesystem Type Size Used Avail Use% Mounted on
svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg nfs4 190G 9.1G 181G 5% /mnt/fsxn/fg
テストファイル作成後、ボリュームの状態を確認します。fg__0001というコンスティチュエントにほとんどのファイルが保存されていることが分かります。
::*> volume show -volume fg* -fields available, filesystem-size, total, used, percent-used, performance-tier-inactive-user-data, performance-tier-inactive-user-data-percent, size, dedupe-space-saved, dedupe-space-saved-percent, dedupe-space-shared,logical-used, logical-used-percent,logical-used-by-afs, logical-available, physical-used, physical-used-percent, tiering-policy
vserver volume size available filesystem-size total used percent-used dedupe-space-saved dedupe-space-saved-percent dedupe-space-shared physical-used physical-used-percent logical-used logical-used-percent logical-available logical-used-by-afs tiering-policy performance-tier-inactive-user-data performance-tier-inactive-user-data-percent
------- ------ ----- --------- --------------- ----- ------ ------------ ------------------ -------------------------- ------------------- ------------- --------------------- ------------ -------------------- ----------------- ------------------- -------------- ----------------------------------- -------------------------------------------
svm fg 200GB 180.9GB 200GB 190GB 9.06GB 4% 0B 0% 0B 9.06GB 5% 9.06GB 5% - 9.06GB none 0B 0%
svm fg__0001
100GB 86.92GB 100GB 95GB 8.08GB 8% 0B 0% 0B 8.08GB 8% 8.08GB 9% - 8.08GB none 0B 0%
svm fg__0002
100GB 94.02GB 100GB 95GB 1005MB 1% 0B 0% 0B 1005MB 1% 1005MB 1% - 1005MB none 0B 0%
3 entries were displayed.
リバランシング
リバランシングをします。
いくつか注意点があるので気をつけながら実行しましょう。Snapshotや重複排除周りは特に気になりますね。リバランシングは頻繁にやるものではなさそうです。
- FlexVol からFlexGroup への変換
FlexVol からFlexGroup への変換後は、FlexGroup の自動リバランシングを使用しないことを推奨します。代わりに、ONTAP 9.10.1以降で使用可能なシステム停止を伴う逆アクティブファイル移動機能を使用するには、を入力します volume rebalance file-move コマンドを実行しますコマンド構文については、を参照してください volume rebalance file-move start のマニュアルページ。
FlexGroupの自動リバランシング機能を使用したリバランシングでは、FlexVolからFlexGroupへの変換を実行し、FlexVolボリューム上のデータの50~85%が新しいコンスティチュエントに移動されるなど、多数のファイルを移動する際のパフォーマンスが低下する可能性があります。- ファイルの最小サイズと最大サイズ
LIFの自動リバランシングで選択されるファイルは、保存されたブロックに基づいています。 リバランシングのために考慮される最小ファイルサイズはデフォルトで100MB（下記のmin-file-sizeパラメータを使用して20MBまで設定可能）で、最大ファイルサイズは100GBです。- Snapshotコピー内のファイル
FlexGroup のリバランシングを設定して、Snapshotコピーに現在存在しないファイルのみを移動することができます。 リバランシングが開始されると、リバランシング処理中にいつでもSnapshotコピー処理がスケジュールされているかどうかが通知されます。
ファイルの移動中で、デスティネーションでフレーミングが実行されている場合、Snapshotコピーは制限されます。 ファイルのリバランシングが実行中の場合、Snapshotコピーのリストア処理は実行できません。- SnapMirror 処理
FlexGroup のリバランシングは、スケジュールされたSnapMirror処理の間に行う必要があります。SnapMirror処理の開始前にファイルを再配置している場合、そのファイルの移動が24分間のSnapMirror再試行期間内に完了しないと、SnapMirror処理が失敗することがあります。 SnapMirror転送の開始後に開始される新しいファイルの再配置は失敗しません。- ファイルベースの圧縮のストレージ効率化
ファイルベースの圧縮によるストレージ効率化では、ファイルはデスティネーションに移動する前に解凍されるため、圧縮による削減が失われます。リバランシング後に手動で開始したバックグラウンドスキャナをFlexGroup で実行した場合、圧縮による削減効果が再び得られます。 ただし、いずれかのボリューム上のSnapshotコピーに関連付けられたファイルは、圧縮の対象として無視されます。- 重複排除
重複排除されたファイルを移動すると、原因 でFlexGroup ボリュームの全体的な使用量が増加する可能性がファイルのリバランシング時には、一意のブロックのみがデスティネーションに移動され、ソースの容量が解放されます。 共有ブロックはソースに保持され、デスティネーションにコピーされます。 このため、ほぼフルのソースコンスティチュエントで使用済み容量を減らすことは目標ですが、新しいデスティネーションに共有ブロックがコピーされるため、FlexGroup ボリューム全体の使用量が増加することもあります。 また、Snapshotコピーの一部であるファイルを移動する場合にも使用できます。Snapshotコピースケジュールがリサイクルされるまでスペース削減は完全には認識されず、Snapshotコピー内のファイルのコピーも削除されます。- FlexClone ボリューム
FlexCloneボリュームの作成時にファイルのリバランシングが実行中の場合、FlexCloneボリュームではリバランシングは実行されません。FlexCloneボリュームでのリバランシングは、FlexCloneボリュームの作成後に実行する必要があります。- ファイル移動
FlexGroup のリバランシング処理中にファイルが移動されると、ソースとデスティネーションの両方のコンスティチュエントについて、クォータアカウンティングの一部としてファイルサイズが報告されます。 移動が完了すると、クォータアカウンティングは通常に戻り、ファイルサイズは新しいデスティネーションでのみ報告されます。- 自律的なランサムウェア防御
ONTAP 9.13.1以降では、システムの停止を伴うリバランシング処理と無停止のリバランシング処理で自律型ランサムウェア対策がサポートされます。- オブジェクトストアボリューム
ボリューム容量のリバランシングは、S3バケットなどのオブジェクトストアボリュームではサポートされていません。
リバランシング前に、リバランシングの状態と設定を確認します。
::*> volume rebalance show
Vserver: svm
Used Size
For Used by Target Imbalance
Volume State Total Imbalance Snapshots Used Size %
----------- -------------- -------- --------- -------- -------- -------- -----
fg not-running 200GB 8.59GB 0B - 3.55GB 41%
fg__0001 not-running 100GB 7.84GB 0B 4.29GB 3.55GB 82%
fg__0002 not-running 100GB 762.6MB 0B 4.29GB -3.55GB -82%
3 entries were displayed.
::*> volume rebalance show -config
Vserver: svm
Max Threshold Max Min Exclude
Volume Runtime Min Max File Moves File Size Snapshot
--------------- ------------ ----- ----- ---------- --------- ---------
fg 6h0m0s 5% 20% 25 100MB true
fg__0001 6h0m0s 5% 20% 25 100MB true
fg__0002 6h0m0s 5% 20% 25 100MB true
3 entries were displayed.
::*> volume rebalance show -instance
Vserver Name: svm
Volume Name: fg
Is Constituent: false
Rebalance State: not-running
Rebalance Notice Messages: -
Total Size: 200GB
Used Size for Calculating Imbalance: 8.59GB
Volume Size Used by Snapshot Copies: 332KB
Constituent Target Used Size: -
Imbalance Size: 3.55GB
Imbalance Percentage: 41%
Moved Data Size: -
Maximum Constituent Imbalance Percentage: 82%
Rebalance Start Time: -
Rebalance Stop Time: -
Rebalance Runtime: -
Rebalance Maximum Runtime: 6h0m0s
Maximum Imbalance Threshold per Constituent: 20%
Minimum Imbalance Threshold per Constituent: 5%
Maximum Concurrent File Moves per Constituent: 25
Minimum File Size: 100MB
Exclude Files Stuck in Snapshot Copies: true
Most Recent File Move Timestamp: -
File Moves Started: 0
Files Scanned: 0
Small Files Skipped: 0
Large Files Skipped: 0
Fast Truncate Files Skipped: 0
Files in Snapshot Copies Skipped: 0
Files Skipped Due to Efficiency (by Block Count): 0
Files Skipped Due to Efficiency (by Percent): 0
Incompatible Files Skipped: 0
Metadata Files Skipped: 0
Remote Caches Skipped: 0
Write Fenced Files Skipped: 0
On Demand Destination Files Skipped: 0
Invalid Footprint Files Skipped: 0
All Other Files Skipped: 0
Blocks Scanned: 0
Blocks Skipped Due to Small Files: 0
Blocks Skipped Due to Large Files: 0
Blocks Skipped Due to Fast Truncate Files: 0
Blocks Skipped Due to Files in Snapshot Copies: 0
Blocks Skipped Due to Efficiency (by Block Count): 0
Blocks Skipped Due to Efficiency (by Percent): 0
Blocks Skipped Due to Incompatible Files: 0
Blocks Skipped Due to Metadata Files: 0
Blocks Skipped Due to Remote Caches: 0
Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
Blocks Skipped Due to Invalid Footprint Files: 0
Blocks Skipped Due to All Other Reasons: 0
Last File Move Error Timestamp: -
Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
Last File Move Error Code: -
Vserver Name: svm
Volume Name: fg__0001
Is Constituent: true
Rebalance State: not-running
Rebalance Notice Messages: -
Total Size: 100GB
Used Size for Calculating Imbalance: 7.84GB
Volume Size Used by Snapshot Copies: 168KB
Constituent Target Used Size: 4.29GB
Imbalance Size: 3.55GB
Imbalance Percentage: 82%
Moved Data Size: -
Maximum Constituent Imbalance Percentage: -
Rebalance Start Time: -
Rebalance Stop Time: -
Rebalance Runtime: -
Rebalance Maximum Runtime: 6h0m0s
Maximum Imbalance Threshold per Constituent: 20%
Minimum Imbalance Threshold per Constituent: 5%
Maximum Concurrent File Moves per Constituent: 25
Minimum File Size: 100MB
Exclude Files Stuck in Snapshot Copies: true
Most Recent File Move Timestamp: -
File Moves Started: 0
Files Scanned: 0
Small Files Skipped: 0
Large Files Skipped: 0
Fast Truncate Files Skipped: 0
Files in Snapshot Copies Skipped: 0
Files Skipped Due to Efficiency (by Block Count): 0
Files Skipped Due to Efficiency (by Percent): 0
Incompatible Files Skipped: 0
Metadata Files Skipped: 0
Remote Caches Skipped: 0
Write Fenced Files Skipped: 0
On Demand Destination Files Skipped: 0
Invalid Footprint Files Skipped: 0
All Other Files Skipped: 0
Blocks Scanned: 0
Blocks Skipped Due to Small Files: 0
Blocks Skipped Due to Large Files: 0
Blocks Skipped Due to Fast Truncate Files: 0
Blocks Skipped Due to Files in Snapshot Copies: 0
Blocks Skipped Due to Efficiency (by Block Count): 0
Blocks Skipped Due to Efficiency (by Percent): 0
Blocks Skipped Due to Incompatible Files: 0
Blocks Skipped Due to Metadata Files: 0
Blocks Skipped Due to Remote Caches: 0
Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
Blocks Skipped Due to Invalid Footprint Files: 0
Blocks Skipped Due to All Other Reasons: 0
Last File Move Error Timestamp: -
Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
Last File Move Error Code: -
Vserver Name: svm
Volume Name: fg__0002
Is Constituent: true
Rebalance State: not-running
Rebalance Notice Messages: -
Total Size: 100GB
Used Size for Calculating Imbalance: 762.6MB
Volume Size Used by Snapshot Copies: 164KB
Constituent Target Used Size: 4.29GB
Imbalance Size: -3.55GB
Imbalance Percentage: -82%
Moved Data Size: -
Maximum Constituent Imbalance Percentage: -
Rebalance Start Time: -
Rebalance Stop Time: -
Rebalance Runtime: -
Rebalance Maximum Runtime: 6h0m0s
Maximum Imbalance Threshold per Constituent: 20%
Minimum Imbalance Threshold per Constituent: 5%
Maximum Concurrent File Moves per Constituent: 25
Minimum File Size: 100MB
Exclude Files Stuck in Snapshot Copies: true
Most Recent File Move Timestamp: -
File Moves Started: 0
Files Scanned: 0
Small Files Skipped: 0
Large Files Skipped: 0
Fast Truncate Files Skipped: 0
Files in Snapshot Copies Skipped: 0
Files Skipped Due to Efficiency (by Block Count): 0
Files Skipped Due to Efficiency (by Percent): 0
Incompatible Files Skipped: 0
Metadata Files Skipped: 0
Remote Caches Skipped: 0
Write Fenced Files Skipped: 0
On Demand Destination Files Skipped: 0
Invalid Footprint Files Skipped: 0
All Other Files Skipped: 0
Blocks Scanned: 0
Blocks Skipped Due to Small Files: 0
Blocks Skipped Due to Large Files: 0
Blocks Skipped Due to Fast Truncate Files: 0
Blocks Skipped Due to Files in Snapshot Copies: 0
Blocks Skipped Due to Efficiency (by Block Count): 0
Blocks Skipped Due to Efficiency (by Percent): 0
Blocks Skipped Due to Incompatible Files: 0
Blocks Skipped Due to Metadata Files: 0
Blocks Skipped Due to Remote Caches: 0
Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
Blocks Skipped Due to Invalid Footprint Files: 0
Blocks Skipped Due to All Other Reasons: 0
Last File Move Error Timestamp: -
Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
Last File Move Error Code: -
3 entries were displayed.
リバランシング後の見込みボリューム使用量が表示されるのはありがたいですね。
リバランシングをします。
::*> volume rebalance start -volume fg
Error: command failed: Failed to start volume capacity rebalancing for volume "fg" in Vserver "svm". Reason: The next scheduled Snapshot copy for volume "fg" in Vserver "svm" is scheduled for Sun Apr 7
08:05:00 2024, in 0h58m33s. This is within the specified 6h0m0s maximum runtime for the volume capacity rebalancing operation. To run the operation, either reduce the "-max-runtime" or disable the
Snapshot policy for the volume.
リバランシングの最大実行時間に対して、Snapshotの取得間隔が短いためエラーとなりました。
Snapshotの取得間隔が6時間未満であることはよくあると思います。実際にリバランシングの運用をする際には最大実行時間をデフォルトの6時間から調整すると良さそうです。
今回はSnapshotポリシーをnoneに変更して、定期的にSnapshotを取得しないようにします。
::*> volume modify -volume fg -snapshot-policy none
Warning: You are changing the Snapshot policy on volume "fg" to "none". Snapshot copies on this volume that do not match any of the prefixes of the new Snapshot policy will not be deleted. However, when the
new Snapshot policy takes effect, depending on the new retention count, any existing Snapshot copies that continue to use the same prefixes might be deleted. See the 'volume modify' man page for more
information.
Do you want to continue? {y|n}: y
[Job 61] Job succeeded: volume modify succeeded
再度リバランシングをしようとします。
::*> volume rebalance start -volume fg
Warning: Granular data will be enabled for volume "fg" in Vserver "svm". Once enabled, this property can only be disabled by restoring a Snapshot copy. Earlier versions of ONTAP are not compatible with this
property, therefore revert to an earlier version of ONTAP will not be possible unless this volume is deleted or restored to a Snapshot copy that was taken before this property was enabled.
Do you want to continue? {y|n}:
今度はGranular dataを有効化すると警告されました。
Granular Dataはリバランシングをするためには必ず有効にする必要があるようです。
Granular Data
ONTAP 9.12.1 introduced the Granular Data management feature for FlexGroups. This feature allows files on a FlexGroup to be moved non-disruptively from one constituent to another.
Granular Data is primarily used by FlexGroup rebalancing. Files that are moved during the rebalancing process will gain an additional management layer just as with remote hardlinks. A file can have both the remote hardlink and Granular Data layers. The rebalancing feature has controls that can limit what kind of files will be chosen for rebalancing. Depending on the settings used for rebalancing, there can be many files that gain the additional layer. Like with remote hardlinks, a file that has this layer will increase the scan time.
TR-4867: Best Practice Guidelines for ONTAP FIle System Analytics
確かにGranular Dataはデフォルトでは有効ではないですね。
::*> volume show -volume fg* -fields granular-data, is-autobalance-eligible
vserver volume is-autobalance-eligible granular-data
------- ------ ----------------------- -------------
svm fg false disabled
svm fg__0001
true disabled
svm fg__0002
true disabled
3 entries were displayed.
特に追加コストがかかったり、パフォーマンスに影響が出たりという記載は見つけられませんでした。
気を取り直して、再度リバランシングをします。
::*> volume rebalance start -volume fg
Warning: Granular data will be enabled for volume "fg" in Vserver "svm". Once enabled, this property can only be disabled by restoring a Snapshot copy. Earlier versions of ONTAP are not compatible with this
property, therefore revert to an earlier version of ONTAP will not be possible unless this volume is deleted or restored to a Snapshot copy that was taken before this property was enabled.
Do you want to continue? {y|n}: y
リバランシング中の状態を確認します。
::*> volume rebalance show -instance
Vserver Name: svm
Volume Name: fg
Is Constituent: false
Rebalance State: rebalancing
Rebalance Notice Messages: -
Total Size: 200GB
Used Size for Calculating Imbalance: 8.59GB
Volume Size Used by Snapshot Copies: 332KB
Constituent Target Used Size: -
Imbalance Size: 3.55GB
Imbalance Percentage: 41%
Moved Data Size: -
Maximum Constituent Imbalance Percentage: 82%
Rebalance Start Time: Sun Apr 07 07:16:47 2024
Rebalance Stop Time: -
Rebalance Runtime: 0h0m37s
Rebalance Maximum Runtime: 6h0m0s
Maximum Imbalance Threshold per Constituent: 20%
Minimum Imbalance Threshold per Constituent: 5%
Maximum Concurrent File Moves per Constituent: 25
Minimum File Size: 100MB
Exclude Files Stuck in Snapshot Copies: true
Most Recent File Move Timestamp: -
File Moves Started: 0
Files Scanned: 90
Small Files Skipped: 0
Large Files Skipped: 0
Fast Truncate Files Skipped: 0
Files in Snapshot Copies Skipped: 90
Files Skipped Due to Efficiency (by Block Count): 0
Files Skipped Due to Efficiency (by Percent): 0
Incompatible Files Skipped: 0
Metadata Files Skipped: 0
Remote Caches Skipped: 0
Write Fenced Files Skipped: 0
On Demand Destination Files Skipped: 0
Invalid Footprint Files Skipped: 0
All Other Files Skipped: 0
Blocks Scanned: 5921640
Blocks Skipped Due to Small Files: 0
Blocks Skipped Due to Large Files: 0
Blocks Skipped Due to Fast Truncate Files: 0
Blocks Skipped Due to Files in Snapshot Copies: 5921640
Blocks Skipped Due to Efficiency (by Block Count): 0
Blocks Skipped Due to Efficiency (by Percent): 0
Blocks Skipped Due to Incompatible Files: 0
Blocks Skipped Due to Metadata Files: 0
Blocks Skipped Due to Remote Caches: 0
Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
Blocks Skipped Due to Invalid Footprint Files: 0
Blocks Skipped Due to All Other Reasons: 0
Last File Move Error Timestamp: -
Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
Last File Move Error Code: -
Vserver Name: svm
Volume Name: fg__0001
Is Constituent: true
Rebalance State: scanning
Rebalance Notice Messages: -
Total Size: 100GB
Used Size for Calculating Imbalance: 7.84GB
Volume Size Used by Snapshot Copies: 168KB
Constituent Target Used Size: 4.29GB
Imbalance Size: 3.55GB
Imbalance Percentage: 82%
Moved Data Size: 0B
Maximum Constituent Imbalance Percentage: -
Rebalance Start Time: Sun Apr 07 07:16:47 2024
Rebalance Stop Time: -
Rebalance Runtime: 0h0m37s
Rebalance Maximum Runtime: 6h0m0s
Maximum Imbalance Threshold per Constituent: 20%
Minimum Imbalance Threshold per Constituent: 5%
Maximum Concurrent File Moves per Constituent: 25
Minimum File Size: 100MB
Exclude Files Stuck in Snapshot Copies: true
Most Recent File Move Timestamp: -
File Moves Started: 0
Files Scanned: 90
Small Files Skipped: 0
Large Files Skipped: 0
Fast Truncate Files Skipped: 0
Files in Snapshot Copies Skipped: 90
Files Skipped Due to Efficiency (by Block Count): 0
Files Skipped Due to Efficiency (by Percent): 0
Incompatible Files Skipped: 0
Metadata Files Skipped: 0
Remote Caches Skipped: 0
Write Fenced Files Skipped: 0
On Demand Destination Files Skipped: 0
Invalid Footprint Files Skipped: 0
All Other Files Skipped: 0
Blocks Scanned: 5921640
Blocks Skipped Due to Small Files: 0
Blocks Skipped Due to Large Files: 0
Blocks Skipped Due to Fast Truncate Files: 0
Blocks Skipped Due to Files in Snapshot Copies: 5921640
Blocks Skipped Due to Efficiency (by Block Count): 0
Blocks Skipped Due to Efficiency (by Percent): 0
Blocks Skipped Due to Incompatible Files: 0
Blocks Skipped Due to Metadata Files: 0
Blocks Skipped Due to Remote Caches: 0
Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
Blocks Skipped Due to Invalid Footprint Files: 0
Blocks Skipped Due to All Other Reasons: 0
Last File Move Error Timestamp: -
Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
Last File Move Error Code: -
Vserver Name: svm
Volume Name: fg__0002
Is Constituent: true
Rebalance State: idle
Rebalance Notice Messages: -
Total Size: 100GB
Used Size for Calculating Imbalance: 762.6MB
Volume Size Used by Snapshot Copies: 164KB
Constituent Target Used Size: 4.29GB
Imbalance Size: -3.55GB
Imbalance Percentage: -82%
Moved Data Size: 0B
Maximum Constituent Imbalance Percentage: -
Rebalance Start Time: Sun Apr 07 07:16:47 2024
Rebalance Stop Time: -
Rebalance Runtime: 0h0m38s
Rebalance Maximum Runtime: 6h0m0s
Maximum Imbalance Threshold per Constituent: 20%
Minimum Imbalance Threshold per Constituent: 5%
Maximum Concurrent File Moves per Constituent: 25
Minimum File Size: 100MB
Exclude Files Stuck in Snapshot Copies: true
Most Recent File Move Timestamp: -
File Moves Started: 0
Files Scanned: 0
Small Files Skipped: 0
Large Files Skipped: 0
Fast Truncate Files Skipped: 0
Files in Snapshot Copies Skipped: 0
Files Skipped Due to Efficiency (by Block Count): 0
Files Skipped Due to Efficiency (by Percent): 0
Incompatible Files Skipped: 0
Metadata Files Skipped: 0
Remote Caches Skipped: 0
Write Fenced Files Skipped: 0
On Demand Destination Files Skipped: 0
Invalid Footprint Files Skipped: 0
All Other Files Skipped: 0
Blocks Scanned: 0
Blocks Skipped Due to Small Files: 0
Blocks Skipped Due to Large Files: 0
Blocks Skipped Due to Fast Truncate Files: 0
Blocks Skipped Due to Files in Snapshot Copies: 0
Blocks Skipped Due to Efficiency (by Block Count): 0
Blocks Skipped Due to Efficiency (by Percent): 0
Blocks Skipped Due to Incompatible Files: 0
Blocks Skipped Due to Metadata Files: 0
Blocks Skipped Due to Remote Caches: 0
Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
Blocks Skipped Due to Invalid Footprint Files: 0
Blocks Skipped Due to All Other Reasons: 0
Last File Move Error Timestamp: -
Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
Last File Move Error Code: -
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 8.59GB - 3.55GB 41% - Sun Apr 07 07:16:47 2024 0h4m53s - 0 55268640
svm fg__0001
scanning - 7.84GB 4.29GB 3.55GB 82% 0B Sun Apr 07 07:16:47 2024 0h4m53s - 0 55268640
svm fg__0002
idle - 762.6MB 4.29GB -3.55GB -82% 0B Sun Apr 07 07:16:47 2024 0h4m53s - 0 0
3 entries were displayed.
::*> volume rebalance file-move statistics show
Vserver: svm
Volume: fg
Source Destination
Constituent Files Bytes Files Bytes
------------------- ----------- ----------- ----------- -----------
fg__0001
In Progress: 0 0B 0 0B
Completed: 0 0B 0 0B
fg__0002
In Progress: 0 0B 0 0B
Completed: 0 0B 0 0B
2 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 8.59GB - 3.55GB 41% - Sun Apr 07 07:16:47 2024 6h0m0s - 0 4214233800
svm fg__0001
scanning - 7.84GB 4.29GB 3.55GB 82% 0B Sun Apr 07 07:16:47 2024 6h0m0s - 0 4214233800
svm fg__0002
idle - 762.6MB 4.29GB -3.55GB -82% 0B Sun Apr 07 07:16:47 2024 6h0m0s - 0 0
3 entries were displayed.
::*> volume show -volume fg* -fields available, filesystem-size, total, used, percent-used, performance-tier-inactive-user-data, performance-tier-inactive-user-data-percent, size, dedupe-space-saved, dedupe-space-saved-percent, dedupe-space-shared,logical-used, logical-used-percent,logical-used-by-afs, logical-available, physical-used, physical-used-percent, tiering-policy
vserver volume size available filesystem-size total used percent-used dedupe-space-saved dedupe-space-saved-percent dedupe-space-shared physical-used physical-used-percent logical-used logical-used-percent logical-available logical-used-by-afs tiering-policy performance-tier-inactive-user-data performance-tier-inactive-user-data-percent
------- ------ ----- --------- --------------- ----- ------ ------------ ------------------ -------------------------- ------------------- ------------- --------------------- ------------ -------------------- ----------------- ------------------- -------------- ----------------------------------- -------------------------------------------
svm fg 200GB 180.9GB 200GB 190GB 9.08GB 4% 0B 0% 0B 9.08GB 5% 9.08GB 5% - 9.08GB none 0B 0%
svm fg__0001
100GB 86.90GB 100GB 95GB 8.10GB 8% 0B 0% 0B 8.10GB 8% 8.10GB 9% - 8.10GB none 0B 0%
svm fg__0002
100GB 94.02GB 100GB 95GB 1005MB 1% 0B 0% 0B 1005MB 1% 1005MB 1% - 1005MB none 0B 0%
3 entries were displayed.
結局6時間待ってもリバランシングはされませんでした。
再リバランシング
偏りが足りないのか? と思い、もう少しテストファイルを追加します。
$ for i in {33..48}; do
sudo dd if=/dev/urandom of=/mnt/fsxn/fg/random_pattern_binary_block_${i} bs=1M count=512
done
$ df -hT -t nfs4
Filesystem Type Size Used Avail Use% Mounted on
svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg nfs4 190G 18G 173G 10% /mnt/fsxn/fg
テストファイル追加後のリバランシングの状態を確認します。
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg not-running "The system scanned 64530 files, totalling 15.82TB. 100% of those files, totalling 100% of the size, have data in Snapshot copies. Files with data in Snapshot copies are not moved by default. To have the volume rebalancer move files with data in Snapshot copies, try the operation again using "-exclude-snapshots false"." 16.66GB - 6.57GB 39% - Sun Apr 07 07:16:47 2024 6h0m0s - 0 4245815880
svm fg__0001
not-running - 14.90GB 8.33GB 6.57GB 78% 0B Sun Apr 07 07:16:47 2024 6h0m0s - 0 4245815880
svm fg__0002
not-running - 1.76GB 8.33GB -6.57GB -78% 0B Sun Apr 07 07:16:47 2024 6h0m0s - 0 0
3 entries were displayed.
::*> volume show -volume fg* -fields available, filesystem-size, total, used, percent-used, performance-tier-inactive-user-data, performance-tier-inactive-user-data-percent, size, dedupe-space-saved, dedupe-space-saved-percent, dedupe-space-shared,logical-used, logical-used-percent,logical-used-by-afs, logical-available, physical-used, physical-used-percent, tiering-policy
vserver volume size available filesystem-size total used percent-used dedupe-space-saved dedupe-space-saved-percent dedupe-space-shared physical-used physical-used-percent logical-used logical-used-percent logical-available logical-used-by-afs tiering-policy performance-tier-inactive-user-data performance-tier-inactive-user-data-percent
------- ------ ----- --------- --------------- ----- ------- ------------ ------------------ -------------------------- ------------------- ------------- --------------------- ------------ -------------------- ----------------- ------------------- -------------- ----------------------------------- -------------------------------------------
svm fg 200GB 172.7GB 200GB 190GB 17.25GB 9% 0B 0% 0B 17.25GB 9% 17.25GB 9% - 17.25GB none 0B 0%
svm fg__0001
100GB 79.74GB 100GB 95GB 15.26GB 16% 0B 0% 0B 15.26GB 15% 15.26GB 16% - 15.26GB none 0B 0%
svm fg__0002
100GB 93.00GB 100GB 95GB 2.00GB 2% 0B 0% 0B 2.00GB 2% 2.00GB 2% - 2.00GB none 0B 0%
3 entries were displayed.
100%のデータがSnapshotにデータを保持していると記載されています。リバランシングの対象はSnapshotに現在存在しないファイルのみです。どうやらSnapshotを気づかないうちに取得していたみたいです。
実際に確認すると、確かにSnapshotが取得されていました。Snapshotポリシーをnoneに設定する前に取得されていたようです。
::*> snapshot show -volume fg*
---Blocks---
Vserver Volume Snapshot Size Total% Used%
-------- -------- ------------------------------------- -------- ------ -----
svm fg
hourly.2024-04-07_0705 8.11GB 4% 33%
fg__0001
hourly.2024-04-07_0705 564KB 0% 0%
fg__0002
hourly.2024-04-07_0705 220KB 0% 0%
3 entries were displayed.
テストファイルを追加した後はSnapshotを取得していません。この状態であればリバランシングされるのではないでしょうか。
再度リバランシングをします。
:*> volume rebalance start -volume fg
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 16.66GB - 6.57GB 39% - Mon Apr 08 00:24:00 2024 0h0m5s - 0 0
svm fg__0001
scanning - 14.90GB 8.33GB 6.57GB 78% 0B Mon Apr 08 00:24:00 2024 0h0m5s - 0 0
svm fg__0002
scanning - 1.76GB 8.33GB -6.57GB -78% 0B Mon Apr 08 00:24:00 2024 0h0m5s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 16.66GB - 6.57GB 39% - Mon Apr 08 00:24:00 2024 0h0m13s - 0 3816140
svm fg__0001
scanning - 14.90GB 8.33GB 6.57GB 78% 0B Mon Apr 08 00:24:00 2024 0h0m13s - 0 3816140
svm fg__0002
idle - 1.76GB 8.33GB -6.57GB -78% 0B Mon Apr 08 00:24:00 2024 0h0m13s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 16.86GB - 6.47GB 38% - Mon Apr 08 00:24:00 2024 0h0m19s Mon Apr 08 00:24:18 2024 3 3816140
svm fg__0001
rebalancing-source
- 14.90GB 8.43GB 6.47GB 76% 0B Mon Apr 08 00:24:00 2024 0h0m19s Mon Apr 08 00:24:18 2024 3 3816140
svm fg__0002
rebalancing-dest
- 1.95GB 8.43GB -6.47GB -76% 0B Mon Apr 08 00:24:00 2024 0h0m19s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 19.17GB - 5.32GB 27% - Mon Apr 08 00:24:00 2024 0h0m22s Mon Apr 08 00:24:21 2024 6 3816140
svm fg__0001
rebalancing-source
- 14.90GB 9.58GB 5.32GB 55% 0B Mon Apr 08 00:24:00 2024 0h0m22s Mon Apr 08 00:24:22 2024 7 3816140
svm fg__0002
rebalancing-dest
- 4.27GB 9.58GB -5.32GB -55% 0B Mon Apr 08 00:24:00 2024 0h0m22s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 21.21GB - 4.30GB 20% - Mon Apr 08 00:24:00 2024 0h0m35s Mon Apr 08 00:24:24 2024 9 3816140
svm fg__0001
rebalancing-source
- 14.90GB 10.60GB 4.30GB 40% 0B Mon Apr 08 00:24:00 2024 0h0m35s Mon Apr 08 00:24:24 2024 9 3816140
svm fg__0002
rebalancing-dest
- 6.30GB 10.60GB -4.30GB -40% 0B Mon Apr 08 00:24:00 2024 0h0m35s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 16.92GB - 2.14GB 12% - Mon Apr 08 00:24:00 2024 0h1m37s Mon Apr 08 00:24:24 2024 9 3816140
svm fg__0001
idle - 10.59GB 8.46GB 2.14GB 25% -4.52GB Mon Apr 08 00:24:00 2024 0h1m37s Mon Apr 08 00:24:24 2024 9 3816140
svm fg__0002
idle - 6.32GB 8.46GB -2.14GB -25% 4.52GB Mon Apr 08 00:24:00 2024 0h1m37s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg idle - 16.71GB - 2.03GB 12% - Mon Apr 08 00:24:00 2024 0h1m58s Mon Apr 08 00:24:24 2024 9 3816140
svm fg__0001
idle - 10.38GB 8.35GB 2.03GB 24% -4.52GB Mon Apr 08 00:24:00 2024 0h1m58s Mon Apr 08 00:24:24 2024 9 3816140
svm fg__0002
idle - 6.32GB 8.35GB -2.03GB -24% 4.52GB Mon Apr 08 00:24:00 2024 0h1m58s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 17.21GB - 1.78GB 10% - Mon Apr 08 00:24:00 2024 0h2m31s Mon Apr 08 00:26:31 2024 11 3816140
svm fg__0001
rebalancing-source
- 10.38GB 8.60GB 1.78GB 20% -4.52GB Mon Apr 08 00:24:00 2024 0h2m31s Mon Apr 08 00:26:31 2024 11 3816140
svm fg__0002
rebalancing-dest
- 6.82GB 8.60GB -1.78GB -20% 4.52GB Mon Apr 08 00:24:00 2024 0h2m31s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 17.21GB - 1.78GB 10% - Mon Apr 08 00:24:00 2024 0h2m40s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0001
rebalancing-source
- 10.38GB 8.60GB 1.78GB 20% -4.52GB Mon Apr 08 00:24:00 2024 0h2m40s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0002
rebalancing-dest
- 6.82GB 8.60GB -1.78GB -20% 4.52GB Mon Apr 08 00:24:00 2024 0h2m40s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg rebalancing - 17.36GB - 858.6MB 4% - Mon Apr 08 00:24:00 2024 0h3m6s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0001
idle - 9.52GB 8.68GB 858.6MB 9% -6.02GB Mon Apr 08 00:24:00 2024 0h3m6s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0002
idle - 7.84GB 8.68GB -858.6MB -9% 6.02GB Mon Apr 08 00:24:00 2024 0h3m6s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg idle - 17.36GB - 858.6MB 4% - Mon Apr 08 00:24:00 2024 0h3m10s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0001
idle - 9.52GB 8.68GB 858.6MB 9% -6.02GB Mon Apr 08 00:24:00 2024 0h3m10s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0002
idle - 7.84GB 8.68GB -858.6MB -9% 6.02GB Mon Apr 08 00:24:00 2024 0h3m10s - 0 0
3 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg idle - 16.72GB - 529.7MB 3% - Mon Apr 08 00:24:00 2024 0h5m18s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0001
idle - 8.88GB 8.36GB 529.7MB 6% -6.02GB Mon Apr 08 00:24:00 2024 0h5m18s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0002
idle - 7.84GB 8.36GB -529.7MB -6% 6.02GB Mon Apr 08 00:24:00 2024 0h5m18s - 0 0
3 entries were displayed.
::*> volume rebalance file-move statistics show
Vserver: svm
Volume: fg
Source Destination
Constituent Files Bytes Files Bytes
------------------- ----------- ----------- ----------- -----------
fg__0001
In Progress: 0 0B 0 0B
Completed: 12 6.02GB 0 0B
fg__0002
In Progress: 0 0B 0 0B
Completed: 0 0B 12 6.02GB
2 entries were displayed.
12個のファイルが3分弱でリバランシングされました。
Imbalance PercentageがMaximum Imbalance Threshold per ConstituentとMinimum Imbalance Threshold per Constituentの間なのでこれ以上待ってもリバランシングは進まなさそうです。停止させましょう。
::*> volume rebalance stop -volume fg
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg not-running "The system scanned 44 files, totalling 14.56GB. 68% of those files, totalling 51% of the size, have data in Snapshot copies. Files with data in Snapshot copies are not moved by default. To have the volume rebalancer move files with data in Snapshot copies, try the operation again using "-exclude-snapshots false"." 16.72GB - 529.7MB 3% - Mon Apr 08 00:24:00 2024 0h9m9s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0001
not-running - 8.88GB 8.36GB 529.7MB 6% -6.02GB Mon Apr 08 00:24:00 2024 0h9m9s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0002
not-running - 7.84GB 8.36GB -529.7MB -6% 6.02GB Mon Apr 08 00:24:00 2024 0h9m9s - 0 0
3 entries were displayed.
ファイル単位のリバランシング
リバランシングはファイル単位でも行えます。
8GiBのファイルを作成します。
$ sudo dd if=/dev/urandom of=/mnt/fsxn/fg/random_pattern_binary_block_49 bs=1M count=8192 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB, 8.0 GiB) copied, 58.1159 s, 148 MB/s $ df -hT -t nfs4 Filesystem Type Size Used Avail Use% Mounted on svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg nfs4 190G 26G 165G 14% /mnt/fsxn/fg
fg__0002の使用量が増加しています。fg__0002に保存されたようですね。
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm fg not-running "The system scanned 44 files, totalling 14.56GB. 68% of those files, totalling 51% of the size, have data in Snapshot copies. Files with data in Snapshot copies are not moved by default. To have the volume rebalancer move files with data in Snapshot copies, try the operation again using "-exclude-snapshots false"." 24.79GB - 3.52GB 14% - Mon Apr 08 00:24:00 2024 0h9m9s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0001
not-running - 8.88GB 12.40GB -3.52GB -28% -6.02GB Mon Apr 08 00:24:00 2024 0h9m9s Mon Apr 08 00:26:32 2024 12 3816140
svm fg__0002
not-running - 15.92GB 12.40GB 3.52GB 28% 6.02GB Mon Apr 08 00:24:00 2024 0h9m9s - 0 0
3 entries were displayed.
先ほど作成したファイルをfg__0002からfg__0001にリバランシングします。
::*> volume rebalance file-move start -volume fg -path /random_pattern_binary_block_49 -destination-constituent fg__0001
File move started with operation-uuid b2f920e6-6e38-4916-aa8b-4491da7e5577.
Use the "volume rebalance file-move show" command to view the status of this operation.
::*> volume rebalance file-move statistics show
Vserver: svm
Volume: fg
Source Destination
Constituent Files Bytes Files Bytes
------------------- ----------- ----------- ----------- -----------
fg__0001
In Progress: 0 0B 1 8.03GB
Completed: 12 6.02GB 0 0B
fg__0002
In Progress: 1 8.03GB 0 0B
Completed: 0 0B 12 6.02GB
2 entries were displayed.
::*> volume rebalance file-move show
Vserver: svm
Volume: fg
Path Source Destination Status Progress
-------------------------- ------------- ------------ --------------- --------
/random_pattern_binary_block_49
fg__0002 fg__0001 Preparing 100%
::*> volume rebalance file-move show -instance
Vserver: svm
Volume Name: fg
Destination Constituent Volume: fg__0001
Operation UUID: b2f920e6-6e38-4916-aa8b-4491da7e5577
Destination Constituent Node: FsxId009351b227391d1f1-01
Destination Constituent DSID: 1033
Source Constituent Volume: fg__0002
Source Operation UUID: b2f920e6-6e38-4916-aa8b-4491da7e5577
Source Constituent Node: FsxId009351b227391d1f1-01
Source Constituent DSID: 1034
Operation Status: Data-Transfer
Scanner Progress Percentage: 5%
Last Failure Reason: -
File Path: /random_pattern_binary_block_49
Destination Constituent MSID: 2163179387
Destination Fileid: 143
Destination Generation: 8112694
Source Constituent MSID: 2163179388
Source Fileid: 510
Source Generation: 8026273
Maximum Scanner Speed (per sec): -
Scanner Paused: false
Scanner Status: Data
Scanner Progress: 371.0MB
Scanner Total: 8GB
Data Scanner Priority: low
Elapsed Time: 0h0m17s
Cutover Time: 4s
Is Snapshot Fenced: false
Is Destination Ready: true
Last Failure Time: -
::*> volume rebalance file-move show
Vserver: svm
Volume: fg
Path Source Destination Status Progress
-------------------------- ------------- ------------ --------------- --------
/random_pattern_binary_block_49
fg__0002 fg__0001 Data-Transfer 17%
::*> volume rebalance file-move show
Vserver: svm
Volume: fg
Path Source Destination Status Progress
-------------------------- ------------- ------------ --------------- --------
/random_pattern_binary_block_49
fg__0002 fg__0001 Data-Transfer 31%
::*> volume rebalance file-move show
Vserver: svm
Volume: fg
Path Source Destination Status Progress
-------------------------- ------------- ------------ --------------- --------
/random_pattern_binary_block_49
fg__0002 fg__0001 Data-Transfer 55%
::*> volume rebalance file-move show
Vserver: svm
Volume: fg
Path Source Destination Status Progress
-------------------------- ------------- ------------ --------------- --------
/random_pattern_binary_block_49
fg__0002 fg__0001 Data-Transfer 68%
::*> volume rebalance file-move show
Vserver: svm
Volume: fg
Path Source Destination Status Progress
-------------------------- ------------- ------------ --------------- --------
/random_pattern_binary_block_49
fg__0002 fg__0001 Data-Transfer 100%
::*> volume rebalance file-move show -instance
Vserver: svm
Volume Name: fg
Destination Constituent Volume: fg__0001
Operation UUID: b2f920e6-6e38-4916-aa8b-4491da7e5577
Destination Constituent Node: FsxId009351b227391d1f1-01
Destination Constituent DSID: 1033
Source Constituent Volume: fg__0002
Source Operation UUID: b2f920e6-6e38-4916-aa8b-4491da7e5577
Source Constituent Node: FsxId009351b227391d1f1-01
Source Constituent DSID: 1034
Operation Status: Data-Transfer
Scanner Progress Percentage: 100%
Last Failure Reason: -
File Path: /random_pattern_binary_block_49
Destination Constituent MSID: 2163179387
Destination Fileid: 143
Destination Generation: 8112694
Source Constituent MSID: 2163179388
Source Fileid: 510
Source Generation: 8026273
Maximum Scanner Speed (per sec): -
Scanner Paused: false
Scanner Status: Data
Scanner Progress: 7.98GB
Scanner Total: 8GB
Data Scanner Priority: low
Elapsed Time: 0h1m4s
Cutover Time: 4s
Is Snapshot Fenced: false
Is Destination Ready: true
Last Failure Time: -
完了しました。簡単ですね。特大ファイルが特定できている場合は役立ちそうです。
FlexGroupのコンスティチュエント間のデータ使用量の偏りが原因でパフォーマンスに影響が出ている時に
FlexGroupのリバランシング機能を試してみました。
FlexGroupのコンスティチュエント間のデータ使用量の偏りが原因でパフォーマンスに影響が出ている時に実行しましょう。
スケジューリング機能もありますが、こちらは定期実行するものではなく指定した時間に一回実行するものです。普段の運用に組み込みものというよりかは、必要性に応じてピンポイントで実行するものかと思います。特にSnapshotと相性はよろしくないので注意が必要です。
この記事が誰かの助けになれば幸いです。
以上、AWS事業本部 コンサルティング部の のんピ(@non____97)でした!
