I tried automated cross-account backup of AWS Secrets Manager secrets

Hello, this is Arahira from the Cloud Business Division, Consulting Department (@eiraces).

Have you ever wanted to take backups of your AWS Secrets Manager secrets across accounts?
While native support through AWS Backup would be ideal, this feature doesn't exist at the time of writing.

Today, I tried a solution from aws-samples that's perfect for when you think, "I want to back up to another account, but setting up the mechanism myself is a bit troublesome..." Thank you, aws-samples!

https://github.com/aws-samples/aws-secrets-manager-cross-account-backup

AWS Secrets Manager Cross Account Backup

The mechanism works like this: when a secret is created or updated in the source account, EventBridge detects the API call, and a Lambda function in the backup account retrieves and replicates the secret from the source account.

The backed-up secrets are saved in the backup account with the naming convention <ACCOUNT>/<REGION>/<SOURCE_NAME>.
It's great that you can immediately see which account and region the secret came from.

Test Configuration

Here's a simple diagram of the configuration for this entry. We'll deploy the following structure using CloudFormation (via Service Catalog).

Trying It Out

Following the README in the GitHub repository, we'll proceed in 4 steps.

Step 1. Preparing Resources

In the backup account, clone the repository we'll be using.
Note that it doesn't matter which account you do this in, as long as it can be referenced as described below.

git clone https://github.com/aws-samples/aws-secrets-manager-cross-account-backup.git
cd aws-secrets-manager-cross-account-backup

Next, upload the files in the scripts/ and templates/ directories to an S3 bucket.

aws s3 cp scripts/ s3://<Bucket-Name>/ --recursive
aws s3 cp templates/ s3://<Bucket-Name>/ --recursive

This S3 bucket needs a bucket policy that allows access from both the source account and the backup account.

Apply a policy like the following (requiring s3:GetObject and s3:ListBucket permissions):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<Account-ID-Dst>:root",
          "arn:aws:iam::<Account-ID-Src>:root"
        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<Bucket-Name>",
        "arn:aws:s3:::<Bucket-Name>/*"
      ]
    }
  ]
}

Also, deploy an execution IAM role in the backup account.

aws iam create-role \
    --role-name StackSetAdminRole \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {"Service": "cloudformation.amazonaws.com"},
        "Action": "sts:AssumeRole"
      }]
    }'

aws iam put-role-policy \
  --role-name StackSetAdminRole \
  --policy-name AssumeExecutionRole \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::*:role/StackSetExecutionRole"
    }]
  }'

Next, deploy the following IAM role in both the backup and source accounts.
Permissions are limited to only the services being used. Enter the backup account ID for Account-ID-Dst.

aws iam create-role \
    --role-name StackSetExecutionRole \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {"AWS": "arn:aws:iam::<Account-ID-Dst>:root"},
        "Action": "sts:AssumeRole"
      }]
    }'

aws iam put-role-policy \
    --role-name StackSetExecutionRole \
    --policy-name StackSetExecution \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Action": [
          "cloudformation:*",
          "events:*",
          "iam:*",
          "lambda:*",
          "secretsmanager:*",
          "s3:*",
          "kms:*",
          "sns:*",
          "logs:*"
        ],
        "Resource": "*"
      }]
    }'

Step 2. Deploy the Baseline

In the backup account, deploy the CloudFormation template secrets_manager_backup.baseline.template.yml.

You need to pass the ARN of an IAM user that can operate Service Catalog, so run aws sts get-caller-identity --query 'Arn' --output text and replace accordingly.

aws cloudformation create-stack \
    --stack-name secrets-manager-backup-baseline \
    --template-url https://s3.amazonaws.com/<Bucket-Name>/templates/secrets_manager_backup.baseline.template.yml \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameters ParameterKey=pResourceLocationBucket,ParameterValue=<Bucket-Name> \
      ParameterKey=pServiceCatalogPortfolioAssociationPrincipalARN,ParameterValue=<IAM-Principal-ARN> \
      ParameterKey=pResourceLocationBucketRegion,ParameterValue=ap-northeast-1 \
      ParameterKey=pResourceLocationBucketKeyPrefix,ParameterValue=templates \
    --region ap-northeast-1

Note that the IAM principal only accepts role, user, or group, and I got stuck with a validate error when trying to use an assumed-role, so be careful.

When the CloudFormation console shows CREATE_COMPLETE, you're good to go.

Once stack creation is complete, a "Secrets Manager Backup" product is registered in Service Catalog.

Step 3. Configure Backup Targets in Service Catalog

Select and launch the "Secrets Manager Backup" product from Service Catalog.

Specify the parameters, including the source account ID for backup.

• Bucket Name
  • S3 bucket name for backup
• Bucket Region
  • Bucket region, in this case ap-northeast-1
• Object Prefix
  • Folder containing deployment files; left blank as templates are stored directly in this case
• Source Account Configuration
  • Source account ID and region
• Backup Account Configuration
  • Backup account ID and region
• CloudFormation Stackset Configuration
  • Administration Role Name is StackSetAdminRole (if you haven't changed it in step 1)
  • Execution Role Name is StackSetExecutionRole (if you haven't changed it in step 1)

By the way, Service Catalog provisioning seems to be adopted with the intention of allowing operators to easily add/remove backup target accounts. Not having to touch CloudFormation directly also helps reduce operational mistakes.

The product launched successfully. If you encounter errors here, it might be due to IAM role configuration issues, so please review them.

Step 4. Verification

Create a test secret in the source account.

aws secretsmanager create-secret \
    --name "test/cross-account-backup" \
    --secret-string '{"username":"admin","password":"P@ssw0rd"}' \
    --tags Key=Environment,Value=test \
    --region ap-northeast-1

Note: If the secret doesn't have tags, the Lambda in the backup account might fail.

After a few minutes, check Secrets Manager in the backup account.
If a secret named <ACCOUNT_ID>/ap-northeast-1/test/cross-account-backup has been created, it's successful.

Verify Update Behavior

Let's also check if backups run when the secret is updated.

Execute the following in the source account:

aws secretsmanager update-secret \
  --secret-id "test/cross-account-backup" \
  --secret-string '{"username":"admin","password":"New-P@ssw0rd!"}' \
  --description "Test secret for cross-account backup" \
  --region ap-northeast-1

Note: During updates, if the secret itself doesn't have a Description, the Lambda in the backup account might fail.

After 30 seconds, I confirmed that it was properly reflected.

Notes

Deletion Behavior

One thing to wonder about is what happens to the backup secret when a secret is deleted in the source account.
Since this wasn't specified in the README, I tested it myself.

Here's the command to delete a secret (be careful as it has the force option):

aws secretsmanager delete-secret \
    --secret-id "test/cross-account-backup" \
    --force-delete-without-recovery \
    --region ap-northeast-1

I deleted the secret in the source account, but of course, it wasn't automatically deleted in the backup account.

Cost

The main costs for this solution are as follows:

In environments with many source secrets, the Secrets Manager fees on the backup side could add up.
There are also Lambda execution charges, but those should be negligible.

Conclusion

Secrets Manager secrets might be one of those resources where you're stumped when asked, "How are you backing these up?"
Since they're not covered by AWS Backup like RDS or EBS, using solutions like this to move them to another account can be effective from a DR perspective.

Many aws-samples repositories can be used as-is, so I recommend taking a look if you're facing similar use cases.

By the way, I thought I could also back up to the Osaka region by entering a value for pStackSetBackupAccountReplicationRegions, but it resulted in what appears to be a permissions error (ReplicateSecretToRegions).
Using this option would likely require modifying the IAM deployed by aws-samples, so I've skipped it for now.

I hope this entry helps someone!
This has been Arahira from the Cloud Business Division, Consulting Department!

References

https://github.com/aws-samples/aws-secrets-manager-cross-account-backup