Please use the Terraform Provider Plugin Cache.
This page has been translated by machine translation. View original
Introduction
Hello everyone, this is Akaike.
While organizing my disk space, I ended up discovering a large number of .terraform folders.
So this time, I'll be testing how to suppress this bloat using Provider Plugin Cache.
The Oversized .terraform Folder
It all started when I was cleaning up insufficient disk space.
When I checked the total size of my .terraform folders, a shocking number came up.
find ~/work ~/github ~/project ~/Documents ~/deploy \
-type d -name .terraform -prune -print0 \
| xargs -0 du -sch | tail -1
81G total
81GB.
I've been at Classmethod for about 1.5 years, and this 81GB tells that story. (Probably.)
Even if each one is only a few hundred MB, this is what happens when projects pile up.
Quick Fix
That said, wasting that much disk space is a problem, so let's clean it up.
Deleting .terraform
First, let's just delete everything to clear things up.
(The .terraform folder is regenerated by terraform init, so there's no problem deleting it.)
find ~/work ~/github ~/project ~/Documents ~/deploy \
-type d -name .terraform -prune -exec rm -rf {} +
This freed up 81GB all at once.
Permanent Solution
As mentioned above, we've freed up 81GB for now.
However, if we do nothing, we'll be back in the same situation 1.5 years from now.
So as a fundamental countermeasure, let's set up Provider Plugin Cache.
What is Provider Plugin Cache?
Provider Plugin Cache is a mechanism that stores downloaded provider binaries in a shared cache directory, and creates symbolic links from each working directory pointing to that location.
Since only one copy of the same provider version is needed, there's no longer a need to hold duplicate binaries for each project.
There are two ways to configure it: using the environment variable TF_PLUGIN_CACHE_DIR, or writing it in the CLI configuration file .terraformrc.
We'll try both this time.
Prerequisites: Create the cache folder in advance
For either method, you need to create the folder for storing the cache beforehand.
mkdir -p $HOME/.terraform.d/plugin-cache
Without Configuration
First, for comparison, let's check the behavior without Plugin Cache configured.
We'll use the following main.tf for testing.
It's a simple configuration that specifies the AWS provider and creates one S3 bucket.
terraform {
required_version = ">= 1.15.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
}
}
provider "aws" {
region = "ap-northeast-1"
}
resource "aws_s3_bucket" "example" {
bucket = "example-bucket"
}
Run terraform init in this state.
terraform init
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v6.60.0
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
After init completes, let's check the size of the .terraform folder.
du -h
778M ./.terraform/providers/registry.terraform.io/hashicorp/aws/6.60.0/darwin_arm64
778M ./.terraform/providers/registry.terraform.io/hashicorp/aws/6.60.0
778M ./.terraform/providers/registry.terraform.io/hashicorp/aws
778M ./.terraform/providers/registry.terraform.io/hashicorp
778M ./.terraform/providers/registry.terraform.io
778M ./.terraform/providers
778M ./.terraform
778M .
778MB for just one AWS provider.
This gets copied for every project.
The 81GB from earlier makes a lot of sense now.
Using Environment Variable
Now, let's specify the cache destination with the environment variable TF_PLUGIN_CACHE_DIR.
export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"
Using the same main.tf as before, let's delete .terraform and run init again.
terraform init
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "6.60.0"...
- Installing hashicorp/aws v6.60.0...
- Installed hashicorp/aws v6.60.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Looking at the working directory after init, .terraform is now almost empty.
du -h
0B ./.terraform/providers/registry.terraform.io/hashicorp/aws/6.60.0
0B ./.terraform/providers/registry.terraform.io/hashicorp/aws
0B ./.terraform/providers/registry.terraform.io/hashicorp
0B ./.terraform/providers/registry.terraform.io
0B ./.terraform/providers
0B ./.terraform
8.0K .
778MB became 0B.
The reason is that the actual provider binary is stored in the cache destination, and the working directory only contains a symbolic link pointing to it.
tree -a .terraform/
.terraform/
└── providers
└── registry.terraform.io
└── hashicorp
└── aws
└── 6.60.0
└── darwin_arm64 -> /Users/akaike.haruka/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/6.60.0/darwin_arm64
The cache folder at the link destination properly contains the actual binary.
tree -a ~/.terraform.d/
.terraform.d/
├── checkpoint_cache
├── checkpoint_signature
├── credentials.tfrc.json
└── plugin-cache
└── registry.terraform.io
└── hashicorp
└── aws
└── 6.60.0
└── darwin_arm64
├── LICENSE.txt
└── terraform-provider-aws_v6.60.0_x5
du -h ~/.terraform.d/
791M .terraform.d//plugin-cache/registry.terraform.io/hashicorp/aws/6.60.0/darwin_arm64
791M .terraform.d//plugin-cache/registry.terraform.io/hashicorp/aws/6.60.0
791M .terraform.d//plugin-cache/registry.terraform.io/hashicorp/aws
791M .terraform.d//plugin-cache/registry.terraform.io/hashicorp
791M .terraform.d//plugin-cache/registry.terraform.io
791M .terraform.d//plugin-cache
791M .terraform.d/
Since only one copy of the actual binary is kept on the cache side, the working directory's disk usage won't increase no matter how many projects use this version.
Using .terraformrc
Instead of an environment variable, you can also specify the configuration in a settings file (.terraformrc).
This is recommended if you want it to always be active without depending on shell settings.
Create .terraformrc directly under $HOME and specify plugin_cache_dir.
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
Let's verify that caching works with this approach too.
Since using the same version would make it hard to see the difference, let's try changing the provider version to 6.59.0.
terraform {
required_version = ">= 1.15.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "6.59.0"
}
}
}
provider "aws" {
region = "ap-northeast-1"
}
resource "aws_s3_bucket" "example" {
bucket = "example-bucket"
}
terraform init
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "6.59.0"...
- Installing hashicorp/aws v6.59.0...
- Installed hashicorp/aws v6.59.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
The cache works properly with .terraformrc as well, and a different version has been installed successfully.
The working directory side is 0B again this time.
du -h .
0B ./.terraform/providers/registry.terraform.io/hashicorp/aws/6.59.0
0B ./.terraform/providers/registry.terraform.io/hashicorp/aws
0B ./.terraform/providers/registry.terraform.io/hashicorp
0B ./.terraform/providers/registry.terraform.io
0B ./.terraform/providers
0B ./.terraform
8.0K .
Looking at the cache destination, 6.59.0 and 6.60.0 coexist neatly organized into separate folders by version.
It seems they are organized in a hierarchy of registry/namespace/provider-name/version/platform.
Beautiful...
tree -a ~/.terraform.d/
/Users/akaike.haruka/.terraform.d/
├── checkpoint_cache
├── checkpoint_signature
├── credentials.tfrc.json
└── plugin-cache
└── registry.terraform.io
└── hashicorp
└── aws
├── 6.59.0
│ └── darwin_arm64
│ ├── LICENSE.txt
│ └── terraform-provider-aws_v6.59.0_x5
└── 6.60.0
└── darwin_arm64
├── LICENSE.txt
└── terraform-provider-aws_v6.60.0_x5
du -h ~/.terraform.d/
791M /Users/akaike.haruka/.terraform.d//plugin-cache/registry.terraform.io/hashicorp/aws/6.60.0/darwin_arm64
791M /Users/akaike.haruka/.terraform.d//plugin-cache/registry.terraform.io/hashicorp/aws/6.60.0
791M /Users/akaike.haruka/.terraform.d//plugin-cache/registry.terraform.io/hashicorp/aws/6.59.0/darwin_arm64
791M /Users/akaike.haruka/.terraform.d//plugin-cache/registry.terraform.io/hashicorp/aws/6.59.0
1.5G /Users/akaike.haruka/.terraform.d//plugin-cache/registry.terraform.io/hashicorp/aws
1.5G /Users/akaike.haruka/.terraform.d//plugin-cache/registry.terraform.io/hashicorp
1.5G /Users/akaike.haruka/.terraform.d//plugin-cache/registry.terraform.io
1.5G /Users/akaike.haruka/.terraform.d//plugin-cache
1.5G /Users/akaike.haruka/.terraform.d/
Different versions are naturally cached as separate entities, so this has grown to 1.5GB.
That said, since the cache only needs one copy no matter how many projects use the same version, the more projects you have, the greater the benefit.
Conclusion
That's all for this discussion on saving disk space using Terraform Provider Plugin Cache.
The bloating of .terraform folders is subtle, but it steadily eats away at your disk like a body blow.
By setting up Provider Plugin Cache, you can avoid downloading the same provider version repeatedly, and keep the working directory's disk usage at nearly zero.
The configuration itself is just one line added to either an environment variable or .terraformrc, so there's really no reason not to set it up.
Whether you're just starting out with Terraform or already using it heavily, if you haven't configured it yet, please give Provider Plugin Cache a try.