tfstateをローカルとS3間で移行してみた

たかくに

2022.04.27

この記事は公開されてから1年以上経過しています。情報が古い可能性がありますので、ご注意ください。

こんにちは、AWS事業本部コンサルティング部のたかくに（@takakuni_）です。

今回は、tfstateをローカルとS3間で移行してみようと思います。

デプロイをローカルでやっていたが、運用に入ってtfstateをS3に保存したくなったケースもあるのではないでしょうか。

私自身、もしもの時に備えて今のうちにできるよう備忘録としてやってみようと思います。

フォルダ構成

以下のようなシンプルなフォルダ構成とファイルで、tfstateをローカルとS3間でキャッチボールしていこうと思います。

すでに、terraform init,terraform applyは実行済みとします。

mv_sfstate/
|- .terraform
|- main.tf
|- provier.tf
|- terraform.tfstate

mv_tfstate/main.tf

resource "aws_vpc" "main" {

  cidr_block = "10.0.0.0/16"
  enable_dns_support = true
  enable_dns_hostnames = true

  tags = {
    "Name" = "mv-tfstate-vpc"
  }
}

mv_tfstate/provider.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.1.0"
    }
  }

  # backend "s3" {
  #   bucket = "example-remote-backend-s3" # 仮の値を入れています
  #   key    = "mv-remote-state/terraform.tfstate"
  #   region = "ap-northeast-1"
  # }
}

provider "aws" {
  # Configuration options
  region = "ap-northeast-1"
}

mv_tfstate/terraform.tfstate

{
  "version": 4,
  "terraform_version": "1.1.7",
  "serial": 5,
  "lineage": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "aws_vpc",
      "name": "main",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "arn": "arn:aws:ec2:ap-northeast-1:XXXXXXXXXXXX:vpc/vpc-XXXXXXXXXXXXXXXXX",
            "assign_generated_ipv6_cidr_block": false,
            "cidr_block": "10.0.0.0/16",
            "default_network_acl_id": "acl-XXXXXXXXXXXXXXXXX",
            "default_route_table_id": "rtb-XXXXXXXXXXXXXXXXX",
            "default_security_group_id": "sg-XXXXXXXXXXXXXXXXX",
            "dhcp_options_id": "dopt-XXXXXXXXXXXXXXXXX",
            "enable_classiclink": false,
            "enable_classiclink_dns_support": false,
            "enable_dns_hostnames": true,
            "enable_dns_support": true,
            "id": "vpc-XXXXXXXXXXXXXXXXX",
            "instance_tenancy": "default",
            "ipv4_ipam_pool_id": null,
            "ipv4_netmask_length": null,
            "ipv6_association_id": "",
            "ipv6_cidr_block": "",
            "ipv6_cidr_block_network_border_group": "",
            "ipv6_ipam_pool_id": "",
            "ipv6_netmask_length": 0,
            "main_route_table_id": "rtb-XXXXXXXXXXXXXXXXX",
            "owner_id": "XXXXXXXXXXXX",
            "tags": {
              "Name": "mv-tfstate-vpc"
            },
            "tags_all": {
              "Name": "mv-tfstate-vpc"
            }
          },
          "sensitive_attributes": [],
          "private": "XXXXXXXXXXXXXXXXXXXX"
        }
      ]
    }
  ]
}

ローカルからS3への移行

tfstateの手動バックアップ(オプション)

後の手順でterraform initを行う際に、自動でterraform.tfstate.backupを作成されます。

「念には念を」の心構えで、手動でもバックアップを取っておくことをオススメします。

takakuni@mv_tfstate % cp -p terraform.tfstate terraform.tfstate.manualbackup

バックエンドの書き換え

はじめにterraformブロックの、backendを変更します。

mv_tfstate/provider.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.1.0"
    }
  }

   backend "s3" {
     bucket = "example-remote-backend-s3" # 仮の値を入れています
     key    = "mv-remote-state/terraform.tfstate"
     region = "ap-northeast-1"
   }
}

provider "aws" {
  # Configuration options
  region = "ap-northeast-1"
}

terraform init

tfstateファイルをS3に送るため、再度terraform initを実行します。

余談ですが、terraform initを実行する前に、terraform planを行うと以下のように「terraform initを実行してね！」とエラーが発生します。

takakuni@mv_tfstate % terraform plan            
╷
│ Error: Backend initialization required, please run "terraform init"
│ 
│ Reason: Initial configuration of the requested backend "s3"
│ 
│ The "backend" is the interface that Terraform uses to store state,
│ perform operations, etc. If this message is showing up, it means that the
│ Terraform configuration you're using is using a custom configuration for
│ the Terraform backend.
│ 
│ Changes to backend configurations require reinitialization. This allows
│ Terraform to set up the new configuration, copy existing state, etc. Please run
│ "terraform init" with either the "-reconfigure" or "-migrate-state" flags to
│ use the current configuration.
│ 
│ If the change reason above is incorrect, please verify your configuration
│ hasn't changed and try again. At this point, no changes to your existing
│ configuration or state have been made.

いざ、terraform init実行

注意すべきポイントは、「Do you want to copy existing state to the new backend?」と対話形式で聞かれているです。

簡単に日本語訳すると、「既存tfstateをS3へコピーしますか？」と聞かれています。

今回はローカルからS3へtfstateを移行したいので「yes」を入力してEnterします。

takakuni@mv_tfstate % terraform init             

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.1.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

tfstateファイルが無事、S3へ移行されていることが確認できました。

terraform planも問題なく実行できました。

takakuni＠mv_tfstate % terraform plan
aws_vpc.main: Refreshing state... [id=vpc-XXXXXXXXXXXXXXXXX]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

移行後のファイルの変化は以下になります。

「mv_tfstate/terraform.tfstate」は空ファイルになる
「mv_tfstate/terraform.tfstate.backup」ファイルの作成
「mv_tfstate/.terraform/terraform.tfstate」ファイルが作成
S3にtfstateファイルの作成

移行が完了したら、mv_tfstate/terraform.tfstateは、空ファイル今後使う用途がない場合は決して問題ありません。

mv_sfstate/
|- .terraform
||- terraform.tfstate # 新規作成
|- main.tf
|- provier.tf
|- terraform.tfstate # ファイルは実在するが何も書き込まれていない
|- terraform.tfstate.backup # 新規作成
|- terraform.tfstate.manualbackup # オプションで作ったファイル

対話形式で「no」を選んだ場合

対話形式で「No」を選んだ場合の挙動は以下の通りです。

「mv_tfstate/terraform.tfstate」は空ファイルになる
「mv_tfstate/terraform.tfstate.backup」ファイルの作成
「mv_tfstate/.terraform/terraform.tfstate」ファイルが作成
S3にtfstateファイルは作成されない

mv_sfstate/
|- .terraform
||- terraform.tfstate # 新規作成
|- main.tf
|- provier.tf
|- terraform.tfstate # ファイルは実在するが何も書き込まれていない
|- terraform.tfstate.backup # 新規作成
|- terraform.tfstate.manualbackup # オプションで作ったファイル

terraform.tfstateは空ファイルになり、S3にもtfstateファイルが作成されていないとプチパニックになりそうですが安心してください。復旧できます。

「terraform.tfstate.backup」または、「terraform.tfstate.manualbackup」を使用して、tfstateファイルをS3上に作ります。

以下のterraform state pushでtfstateファイルを復旧します。

# 「terraform.tfstate.backup」または、「terraform.tfstate.manualbackup」
takakuni@mv_tfstate % terraform state push terraform.tfstate.backup
takakuni@mv_tfstate % terraform  plan

aws_vpc.main: Refreshing state... [id=vpc-XXXXXXXXXXXXXXXXX]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

.terraform/terraform.tfstateとは

「.terraform/terraform.tfstate」について解説します。

このファイルは、リモートステートの向き先を指定するためのファイルとなります。

「terraform.tfstate」とファイル名がとても似ていますが削除しても復旧可能です。

.terraform/terraform.tfstate

{
    "version": 3,
    "serial": 1,
    "lineage": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "backend": {
        "type": "s3",
        "config": {
            "access_key": null,
            "acl": null,
            "assume_role_duration_seconds": null,
            "assume_role_policy": null,
            "assume_role_policy_arns": null,
            "assume_role_tags": null,
            "assume_role_transitive_tag_keys": null,
            "bucket": "example-remote-backend-s3",
            "dynamodb_endpoint": null,
            "dynamodb_table": null,
            "encrypt": null,
            "endpoint": null,
            "external_id": null,
            "force_path_style": null,
            "iam_endpoint": null,
            "key": "mv-remote-state/terraform.tfstate",
            "kms_key_id": null,
            "max_retries": null,
            "profile": null,
            "region": "ap-northeast-1",
            "role_arn": null,
            "secret_key": null,
            "session_name": null,
            "shared_credentials_file": null,
            "skip_credentials_validation": null,
            "skip_metadata_api_check": null,
            "skip_region_validation": null,
            "sse_customer_key": null,
            "sts_endpoint": null,
            "token": null,
            "workspace_key_prefix": null
        },
        "hash": XXXXXXXXXX
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {},
            "depends_on": []
        }
    ]
}

実際に削除から復旧までの流れは以下の通りです。

# 間違えて削除
takakuni@mv_tfstate % rm -f .terraform/terraform.tfstate

# terraform planが通らなくなる
takakuni@mv_tfstate % terraform plan
╷
│ Error: Backend initialization required, please run "terraform init"
│ 
│ Reason: Initial configuration of the requested backend "s3"
│ 
│ The "backend" is the interface that Terraform uses to store state,
│ perform operations, etc. If this message is showing up, it means that the
│ Terraform configuration you're using is using a custom configuration for
│ the Terraform backend.
│ 
│ Changes to backend configurations require reinitialization. This allows
│ Terraform to set up the new configuration, copy existing state, etc. Please run
│ "terraform init" with either the "-reconfigure" or "-migrate-state" flags to
│ use the current configuration.
│ 
│ If the change reason above is incorrect, please verify your configuration
│ hasn't changed and try again. At this point, no changes to your existing
│ configuration or state have been made.
╵

# terraform initの再実行
takakuni＠mv_tfstate % terraform init 

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.1.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

# terraform planが通る
takakuni＠mv_tfstate % terraform plan
aws_vpc.main: Refreshing state... [id=vpc-XXXXXXXXXXXXXXXXX]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

S3からローカルへの移行

逆も然りで、「やっぱ戻して！」と言われた時の為にやってみます。

tfstateの手動バックアップ(オプション)

今回も念の為、tfstateファイルのバックアップを手動で取得します。

S3からローカルへ移行する場合のterraform initではterraform.tfstate.backupが作成されないため、やってて損はない気がします。

takakuni@mv_tfstate % terraform state pull > terraform.tfstate.manualbackupformS3

バックエンドの書き換え

terraformブロックの、backendを変更します。

コメントアウトし直して完了です。

mv_tfstate/provider.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.1.0"
    }
  }

  # backend "s3" {
  #   bucket = "example-remote-backend-s3" # 仮の値を入れています
  #   key    = "mv-remote-state/terraform.tfstate"
  #   region = "ap-northeast-1"
  # }
}

provider "aws" {
  # Configuration options
  region = "ap-northeast-1"
}

terraform initの実行

terraform planを実行すると、「terraform initを実行してね！」と怒られます。

takakuni@mv_tfstate % terraform plan
╷
│ Error: Backend initialization required, please run "terraform init"
│ 
│ Reason: Unsetting the previously set backend "s3"
│ 
│ The "backend" is the interface that Terraform uses to store state,
│ perform operations, etc. If this message is showing up, it means that the
│ Terraform configuration you're using is using a custom configuration for
│ the Terraform backend.
│ 
│ Changes to backend configurations require reinitialization. This allows
│ Terraform to set up the new configuration, copy existing state, etc. Please run
│ "terraform init" with either the "-reconfigure" or "-migrate-state" flags to
│ use the current configuration.
│ 
│ If the change reason above is incorrect, please verify your configuration
│ hasn't changed and try again. At this point, no changes to your existing
│ configuration or state have been made.

また、S3からローカルへ移行する際は、-migrate-stateフラッグを付与して実行する必要があります。

先ほどと一緒で、「既存tfstateの情報を保持するかどうか」の質問に対して「yes」または「no」で入力しEnterを押します。

takakuni@mv_tfstate % terraform init -migrate-state

Initializing the backend...
Terraform has detected you're unconfiguring your previously set "s3" backend.
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "s3" backend to the
  newly configured "local" backend. No existing state was found in the newly
  configured "local" backend. Do you want to copy this state to the new "local"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Successfully unset the backend "s3". Terraform will now operate locally.

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.1.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
# terraform planで差分確認
takakuni@tfstate % terraform plan     
aws_vpc.main: Refreshing state... [id=vpc-XXXXXXXXXXXXXXXXX]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.