Recovered from remote state after Terraform apply failed midway

Introduction

Even when you verify what will be created with Terraform plan, API calls during apply can fail due to insufficient permissions. In this case, resources that were already created are not automatically deleted.

This time, I ran Terraform to create an IAM role and Amazon S3 objects using a work role with restricted permissions. The plan was to add 5 resources, but at the stage of creating 2 S3 objects, s3:PutObjectTagging was denied and apply stopped midway.

This article presents a log of work where, after a Terraform apply failed partway through, I checked which resources had been created and which had not, then fixed only the missing permissions and re-ran the operation. This time, I did not manually delete already-created resources, but instead cross-referenced Terraform's standard output, remote state, and AWS API results. After that, I added only the missing permissions, rebuilt the plan from remote state, and applied it.

Target Audience

• Those whose Terraform apply has failed midway and are unsure how to handle already-created resources
• Those using remote state and saved plans
• Those who want to re-run apply without over-expanding AWS permissions
• Those who want to separate raw Terraform logs from records intended for publication

Verification Environment

References

• HashiCorp: Apply Terraform configuration
• HashiCorp: terraform plan command
• HashiCorp: S3 backend
• HashiCorp: State locking
• HashiCorp: Configure default tags for AWS resources
• AWS: PutObjectTagging

Configuration

The verification target is an IAM role for use with SageMaker Processing Jobs and small input files placed in S3. No GPUs or Processing Jobs are created.

The AWS Provider had common default tags configured.

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      Environment = "poc"
      ManagedBy   = "terraform"
      Project     = "example"
    }
  }
}

default_tags are automatically applied to resources that support tagging. The S3 objects in this case were also planned to have default tags set.

Diagnostic Method

Keeping Terraform's Standard Output as Primary Information

To investigate the cause of the apply failure, I saved both stdout and stderr to a single log file outside Git management. At the same time, I recorded the exit code, log SHA-256, the failed Terraform resource address, AWS API, HTTP status, and AWS error code to JSON.

python scripts/capture_terraform.py `
  --log-file private/apply.log `
  --summary-file private/apply-summary.json `
  --run-id static-apply-001 `
  --plan-sha256 <approved-plan-sha256> `
  -- terraform apply processing-static.tfplan

Raw logs may contain request IDs, ARNs, bucket names, and similar information. These are not added to Git, and only anonymized items are written in records intended for publication.

The JSON took the following form.

{
  "terraform_exit_code": 1,
  "resource_address": "aws_s3_object.driver",
  "aws_service": "S3",
  "api_operation": "PutObject",
  "http_status_code": 403,
  "aws_error_code": "AccessDenied",
  "log_sha256": "<sha256>"
}

Only when the resource and API could not be identified from Terraform's standard output alone would I check supplementary information such as CloudTrail. Since the standard output contained the necessary information this time, CloudTrail was not used.

Verifying Both Remote State and AWS API

At the point of apply failure, the S3 backend state serial had advanced from 0 to 1. The state lock was already released.

The plan was to add 5 resources, but in reality 3 resources were created and 2 were not. I confirmed that the results matched not only in Terraform's error display, but also between remote state and AWS API results.

Actions Taken

Adding Only the Missing Permissions

The error recorded that s3:PutObjectTagging was not permitted. s3:PutObjectTagging is required to set tags on objects.

In this case, I added permissions for the driver and fixture object ARNs.

statement {
  effect  = "Allow"
  actions = ["s3:PutObjectTagging"]
  resources = [
    local.driver_object_arn,
    local.fixture_object_arn,
  ]
}

This was not expanded to the entire input prefix or the entire bucket. I also confirmed that the 2 permitted objects would be allowed and that adjacent unauthorized keys would be implicitDeny.

What could be confirmed here was the missing action and the target objects. I did not make definitive claims from the saved error about the order in which the AWS Provider internally calls S3 APIs.

Discarding the Old Saved Plan

The plan used at the time of failure was built based on state serial 0. The remote state after apply is at serial 1, and the IAM resources already exist.

Therefore, the old plan is not reused. After fixing the permissions, I rebuilt the plan from remote state serial 1.

terraform plan -out=processing-static-recovery.tfplan
terraform show processing-static-recovery.tfplan

A plan saved with terraform plan -out is a file for later apply using the state and inputs at the time of creation. Plan files may contain sensitive values in plaintext. These are not added to Git; only the approved SHA-256 and action count are shared.

The contents of the rebuilt plan were as follows.

There were no changes to the IAM role trust policy or execution policy. I was able to confirm that the state after the failure was reflected in the plan.

Applying the Rebuilt Plan

The approved new plan was applied exactly once.

The 2 objects are stored in a bucket with versioning and SSE-S3 enabled. I retrieved the VersionId, size, Content-Type, SHA-256 metadata, and default tags via the AWS API and confirmed they matched the local files.

Summary

After Terraform apply failed midway, I checked the current state from remote state and standard output, then re-ran with a new plan that added only the missing permissions. Rather than immediately fixing an apply failure with manual operations, starting from the state recorded by Terraform allows you to safely resume while verifying the changes. I hope the insights from this article will be helpful when Terraform apply fails midway.

Appendix

Terraform's standard output and exit code were saved outside Git management using a Python script. The appendix presents the processing necessary for the article, extracted from the code actually used.

from __future__ import annotations

import argparse
import hashlib
import json
import re
import subprocess
from datetime import datetime, timezone
from pathlib import Path

ANSI_ESCAPE = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]")
RESOURCE_PATTERN = re.compile(r"(?m)^\s*with\s+([^,\r\n]+),\s*$")
OPERATION_PATTERN = re.compile(
    r"operation error\s+([^:\r\n]+):\s*([A-Za-z][A-Za-z0-9]+)"
)
STATUS_PATTERN = re.compile(r"StatusCode:\s*(\d{3})")
ERROR_CODE_PATTERN = re.compile(
    r"RequestID:\s*[^,\r\n]+,\s*(?:api error\s+)?([A-Za-z][A-Za-z0-9]+):"
)

def utc_now() -> str:
    return datetime.now(timezone.utc).isoformat()

def atomic_json(path: Path, value: dict) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    temporary = path.with_suffix(path.suffix + ".tmp")
    temporary.write_text(json.dumps(value, indent=2) + "\n", encoding="utf-8")
    temporary.replace(path)

def normalized_output(raw: bytes) -> str:
    text = raw.decode("utf-8", errors="replace")
    text = ANSI_ESCAPE.sub("", text)
    lines = []
    for line in text.splitlines():
        line = re.sub(r"^\s*[╷│╵]\s?", "", line)
        lines.append(line.rstrip())
    return "\n".join(lines)

def extract_diagnostic(raw: bytes) -> dict:
    text = normalized_output(raw)
    resource = RESOURCE_PATTERN.search(text)
    operation = OPERATION_PATTERN.search(text)
    status = STATUS_PATTERN.search(text)
    error_code = ERROR_CODE_PATTERN.search(text)
    first_error = re.search(r"(?m)^Error:\s*(.+)$", text)
    return {
        "terraform_error_summary": (
            first_error.group(1).strip() if first_error else None
        ),
        "resource_address": resource.group(1).strip() if resource else None,
        "aws_service": operation.group(1).strip() if operation else None,
        "api_operation": operation.group(2) if operation else None,
        "http_status_code": int(status.group(1)) if status else None,
        "aws_error_code": error_code.group(1) if error_code else None,
    }

def run_and_capture(
    command: list[str],
    log_path: Path,
    summary_path: Path,
    run_id: str,
    plan_sha256: str,
) -> int:
    started_at = utc_now()
    log_path.parent.mkdir(parents=True, exist_ok=True)
    digest = hashlib.sha256()

    process = subprocess.Popen(
        command,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
    )
    assert process.stdout is not None

    with log_path.open("wb") as handle:
        while chunk := process.stdout.read(65536):
            handle.write(chunk)
            handle.flush()
            digest.update(chunk)

    process.stdout.close()
    exit_code = process.wait()
    diagnostic = extract_diagnostic(log_path.read_bytes())
    summary = {
        "run_id": run_id,
        "plan_sha256": plan_sha256,
        "started_at": started_at,
        "completed_at": utc_now(),
        "terraform_exit_code": exit_code,
        "log_file": str(log_path),
        "log_sha256": digest.hexdigest(),
        **diagnostic,
    }
    summary["cloudtrail_fallback_required"] = exit_code != 0 and (
        not summary["resource_address"] or not summary["api_operation"]
    )
    atomic_json(summary_path, summary)
    return exit_code

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--log-file", type=Path, required=True)
    parser.add_argument("--summary-file", type=Path, required=True)
    parser.add_argument("--run-id", required=True)
    parser.add_argument("--plan-sha256", required=True)
    parser.add_argument("command", nargs=argparse.REMAINDER)
    args = parser.parse_args()
    command = args.command[1:] if args.command[:1] == ["--"] else args.command
    if not command:
        parser.error("a Terraform command is required after --")
    return run_and_capture(
        command,
        args.log_file,
        args.summary_file,
        args.run_id,
        args.plan_sha256,
    )

if __name__ == "__main__":
    raise SystemExit(main())