TerraformのCI/CDパイプラインを実装してみた

2022.06.22

こんにちは!AWS事業本部コンサルティング部のたかくにです。

今回は、TerraformのCI/CDパイプラインでどのような構成が取れるか考えてみました。

きっかけは、最近tfsecを使う機会がよくあり、CI/CDに組み込むと面白そうだなと思い、GWの宿題として考えてみました。

気がついたら、6月の終盤で超大作ブログになっていました。是非ともTerraformユーザーの方にご覧いただけるととても嬉しいです。

今回の構成が「必ずしも正解」というわけではなく、あくまで一例として参考程度にご覧いただけると幸いです。

「tfsec」って何?と言う方は以下のサイトも合わせてご覧いただけると幸いです。

全体の構成図

以下の構成図のようなTerraform実行パイプラインを作成しようと思います。

パイプラインが少し長いため、以下の区分で実装方式や解説をまとめていこうと思います。

  • CodeCommit、CodeBuild(tfsec)の実装
  • Backend(S3, DynamoDB)の実装
  • CodeBuild(plan)、CodeBuild(apply)の実装
  • CodePipelineで結合

実装はCloudFormationで行います。「Terraform使わずにCloudFormationでIaCやればいいじゃん」はその通りですがご容赦いただければと思います。

最終的なyamlファイルは以下になります。あまりにも長いため折り返しています。

ファイル全体(クリックで表示できます)

terraform_pipeline.yml

AWSTemplateFormatVersion: '2010-09-09'
Description: Terraform pipeline template.
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    - Label:
        default: "Project Name Prefix"
      Parameters:
        - PrjPrefix
    - Label:
        default: "Repository Configuration"
      Parameters:
        - BranchName
    - Label:
        default: "Docker Login Profile"
      Parameters:
      - DockerHubUserName
      - DockerHubUserPassword
    - Label:
        default: "Logs Configuration"
      Parameters:
        - BuildLogsRationDay
    - Label:
        default: "terraform build configuration"
      Parameters:
        - TerraformVersion

Parameters:
  PrjPrefix:
    Type: String
    Description: "Enter resource prefix. (ex. terraform-dev)"
    Default: "terraform-dev"
  BranchName:
    Type: String
    Description: "Enter the name of the branch where you want to store exclude.yml"
    Default: "main"
  DockerHubUserName:
    Type: String
    Description: "Enter Docker Hub loging user name. (ex. terraform)"
    NoEcho: true
  DockerHubUserPassword:
    Type: String
    Description: "Enter Docker Hub loging user password. (ex. P@ssw0rd) If you enable MFA in Docker Hub, generate AccessToken and enter instead of user password."
    NoEcho: true
  BuildLogsRationDay:
    Type: Number
    Description: "Enter build log retention period."
    Default: 90
    AllowedValues:
      - 1
      - 3
      - 5
      - 7
      - 14
      - 30
      - 60
      - 90
      - 120
      - 150
      - 180
      - 365
      - 400
      - 545
      - 731
      - 1827
      - 3653
  TerraformVersion:
    Type: String
    Description: "Enter the version of Terraform that use with CodeBuild."
    Default: "1.2.0"

Resources:
#################################
# KMS (CloudWatch Logs)
#################################
  KeyCWL:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline Build Logs Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
          - Sid: Allow use of the key from CWL
            Effect: "Allow"
            Principal:
              Service: !Sub "logs.${AWS::Region}.amazonaws.com"
            Action:
              - "kms:Encrypt"
              - "kms:Decrypt"
              - "kms:ReEncrypt"
              - "kms:GenerateDataKey"
              - "kms:Describe"
            Resource: "*"
            Condition:
              ArnLike:
                kms:EncryptionContext:aws:logs:arn: !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-logs-key"

  KeyAliasCWL:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-logs-key"
      TargetKeyId: !Ref KeyCWL

#################################
# KMS (S3 arthifact)
#################################
  KeyS3Arthifact:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline arthifact Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-artifact-key"

  KeyAliasS3Arthifact:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-artifact-key"
      TargetKeyId: !Ref KeyS3Arthifact

#################################
# KMS (S3 tfstate)
#################################
  KeyS3Tfstate:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline tfstate Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-tfstate-key"

  KeyAliasS3Tfstate:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-tfstate-key"
      TargetKeyId: !Ref KeyS3Tfstate

#################################
# KMS (Secrets Manager)
#################################
  KeySecretsManager:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline Secrets Manager Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-secretsmanager-key"

  KeyAliasSecretsManager:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-secretsmanager-key"
      TargetKeyId: !Ref KeySecretsManager

#################################
# KMS (DynamoDB)
#################################
  KeyDynamoDB:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline DynamoDB Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-dynamoDB-key"

  KeyAliasDynamoDB:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-dynamoDB-key"
      TargetKeyId: !Ref KeyDynamoDB

#################################
# CodeCommit
#################################
  CodeCommit:
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName: !Sub "${PrjPrefix}-tf-repo"
      RepositoryDescription: !Sub "${PrjPrefix}-tf-repo"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-repo"

#################################
# Custom Resource
#################################
  PutExcludeRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        - "arn:aws:iam::aws:policy/AWSCodeCommitPowerUser"
      RoleName: !Sub "${PrjPrefix}-tf-repo-${BranchName}-put-exclude-role"

  PutExcludeLog:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/lambda/${PrjPrefix}-tf-repo-${BranchName}-put-exclude"
      KmsKeyId: !GetAtt KeyCWL.Arn

  PutExcludeFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Sub "${PrjPrefix}-tf-repo-${BranchName}-put-exclude"
      Code:
        ZipFile: |
          import json
          import boto3
          import cfnresponse

          def handler(event, context):
              try:
                  repository = event['ResourceProperties']['RepositoryName']
                  branch = event['ResourceProperties']['BranchName']
                  content = event['ResourceProperties']['FileContent'].encode()
                  path = event['ResourceProperties']['FilePath']

                  if event['RequestType'] == 'Create':
                      codecommit = boto3.client('codecommit')
                      response = codecommit.put_file(
                          repositoryName=repository,
                          branchName=branch,
                          fileContent=content,
                          filePath=path,
                          commitMessage='Initial Commit',
                          name='Your Lambda Helper'
                      )
                      cfnresponse.send(event, context, cfnresponse.SUCCESS, response)
                  if event['RequestType'] == 'Delete':
                      cfnresponse.send(event, context, cfnresponse.SUCCESS, {'Response': 'Success'})
                  if event['RequestType'] == 'Update':
                      cfnresponse.send(event, context, cfnresponse.SUCCESS, {'Response': 'Success'})
              except Exception as e:
                  print(e)
                  cfnresponse.send(event, context, cfnresponse.FAILED, {})
      Handler: index.handler
      MemorySize: 128
      Role: !GetAtt PutExcludeRole.Arn
      Runtime: "python3.9"
      Timeout: 60
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-repo-${BranchName}-put-exclude"
    DependsOn: PutExcludeLog

  PutExclude:
    Type: Custom::CodeCommitPutExclude
    Properties:
      ServiceToken: !GetAtt PutExcludeFunction.Arn
      RepositoryName: !GetAtt CodeCommit.Name
      BranchName: !Ref BranchName
      FileContent: "---\nexclude:"
      FilePath: "exclude.yml"

#################################
# Secrets Manager
#################################
  DockerLoginProfile:
    Type: AWS::SecretsManager::Secret
    Properties:
      Description: "Docker login profile for terraform pipelines"
      Name: !Sub "${PrjPrefix}/docker_hub"
      SecretString: !Sub '{"username":"${DockerHubUserName}","password":"${DockerHubUserPassword}"}'
      KmsKeyId: !GetAtt KeySecretsManager.Arn
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}/docker_hub"

#################################
# CloudWatch Logs (CodeBuild tfsec)
#################################
  CWLTfsec:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfsec-project"
      RetentionInDays: !Ref BuildLogsRationDay
      KmsKeyId: !GetAtt KeyCWL.Arn
      Tags: 
        - Key: "Name"
          Value: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfsec-project"

#################################
# S3 Bucket (Artifact)
#################################
  BucketArtifacts:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub "${PrjPrefix}-tf-pipeline-artifacts"
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: "aws:kms"
              KMSMasterKeyID: !GetAtt KeyS3Arthifact.Arn
            BucketKeyEnabled: true
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-artifacts"

  BucketPolicyArtifacts:
      Type: AWS::S3::BucketPolicy
      Properties: 
        Bucket: !Ref BucketArtifacts
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Sid: "AllowSSLRequestsOnly"
              Effect: "Deny"
              Principal: "*"
              Action: "s3:*"
              Resource:
                - !GetAtt BucketArtifacts.Arn
                - !Sub
                  - "${BucketArn}/*"
                  - { BucketArn: !GetAtt BucketArtifacts.Arn }
              Condition:
                Bool:
                  aws:SecureTransport: "false"

#################################
# CodeBuild (tfsec)
#################################
  PolicyTfsec:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${PrjPrefix}-tf-build-tfsec-project-policy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: "BuildLogs"
            Effect: "Allow"
            Action:
              - "logs:CreateLogGroup"
              - "logs:CreateLogStream"
              - "logs:PutLogEvents"
            Resource:
              - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CWLTfsec}"
              - !GetAtt CWLTfsec.Arn
          - Sid: "BuildReports"
            Effect: "Allow"
            Action:
              - "codebuild:CreateReportGroup"
              - "codebuild:CreateReport"
              - "codebuild:UpdateReport"
              - "codebuild:BatchPutTestCases"
              - "codebuild:BatchPutCodeCoverages"
            Resource:
              - !Sub "arn:${AWS::Partition}:codebuild:${AWS::Region}:${AWS::AccountId}:report-group/${PrjPrefix}-tf-build-tfsec-project-reports"
          - Sid: "GitPull"
            Effect: "Allow"
            Action: "codecommit:GitPull"
            Resource: !GetAtt CodeCommit.Arn
          - Sid: "GetSecretValue"
            Effect: "Allow"
            Action: "secretsmanager:GetSecretValue"
            Resource: !Ref DockerLoginProfile
          - Sid: "KmsKey"
            Effect: "Allow"
            Action:
              - "kms:Decrypt"
              - "kms:DescribeKey"
              - "kms:Encrypt"
              - "kms:ReEncrypt"
              - "kms:GenerateDataKey"
            Resource:
              - !GetAtt KeyS3Arthifact.Arn
              - !GetAtt KeySecretsManager.Arn
          - Sid: "S3Artifact"
            Effect: "Allow"
            Action:
              - "s3:PutObject"
              - "s3:GetObject"
              - "s3:GetObjectVersion"
              - "s3:GetBucketAcl"
              - "s3:GetBucketLocation"
            Resource:
              - !GetAtt BucketArtifacts.Arn
              - !Sub
                - "${BucketArn}/*"
                - { BucketArn: !GetAtt BucketArtifacts.Arn }

  RoleTfsec:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-build-tfsec-project-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "codebuild.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - !Ref PolicyTfsec
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfsec-project-role"

  ProjectTfsec:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Sub "${PrjPrefix}-tf-build-tfsec-project"
      Description: "Analyze the code for vulnerabilities using tfsec."
      Source:
        Type: "CODEPIPELINE"
        BuildSpec: |
          version: 0.2
          env:
            exported-variables:
              - BuildID
              - BuildTag
          phases:
            pre_build:
              commands:
                - "echo Executing tfsec"
                - "mkdir -p reports/tfsec/"
            build:
              commands:
                - "tfsec -s --no-color --config-file exclude.yml ."
                - "tfsec -s --no-color --config-file exclude.yml . --format junit > reports/tfsec/report.xml"
            post_build:
              commands:
                - "export BuildID=`echo $CODEBUILD_BUILD_ID | cut -d: -f1`"
                - "export BuildTag=`echo $CODEBUILD_BUILD_ID | cut -d: -f2`"
          reports:
            reports:
              files:
                - "reports/tfsec/report.xml"
              file-format: JUNITXML
      Artifacts:
        Type: "CODEPIPELINE"
      Cache:
        Type: "LOCAL"
        Modes:
          - "LOCAL_DOCKER_LAYER_CACHE"
      Environment:
        Type: "LINUX_CONTAINER"
        ComputeType: "BUILD_GENERAL1_SMALL"
        Image: "aquasec/tfsec:latest"
        ImagePullCredentialsType: "SERVICE_ROLE"
        RegistryCredential:
          Credential: !Ref DockerLoginProfile
          CredentialProvider: "SECRETS_MANAGER"
        PrivilegedMode: true
      LogsConfig:
        CloudWatchLogs:
          Status: "ENABLED"
          GroupName: !Ref CWLTfsec
      EncryptionKey: !GetAtt KeyS3Arthifact.Arn
      ResourceAccessRole: !GetAtt RoleTfsec.Arn
      ServiceRole: !GetAtt RoleTfsec.Arn
      TimeoutInMinutes: 60
      Visibility: "PRIVATE"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfsec-project"

  ReportGroupTfsec:
    Type: AWS::CodeBuild::ReportGroup
    Properties:
      Name: !Sub "${ProjectTfsec}-reports"
      Type: "TEST"
      DeleteReports: true
      ExportConfig:
        ExportConfigType: "NO_EXPORT"
      Tags: 
        - Key: "Name"
          Value: !Sub "${ProjectTfsec}-reports"

#################################
# S3 Bucket (tfstate)
#################################
  BucketTfstate:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub "${PrjPrefix}-tf-pipeline-tfstate"
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: "aws:kms"
              KMSMasterKeyID: !GetAtt KeyS3Tfstate.Arn
            BucketKeyEnabled: true
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: "Enabled"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-tfstate"

  BucketPolicyTfstate:
      Type: AWS::S3::BucketPolicy
      Properties: 
        Bucket: !Ref BucketTfstate
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Sid: "AllowSSLRequestsOnly"
              Effect: "Deny"
              Principal: "*"
              Action: "s3:*"
              Resource:
                - !GetAtt BucketTfstate.Arn
                - !Sub
                  - "${BucketArn}/*"
                  - { BucketArn: !GetAtt BucketTfstate.Arn }
              Condition:
                Bool:
                  aws:SecureTransport: "false"

#################################
# DynamoDB table (state lock)
#################################
  DDBTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Sub "${PrjPrefix}-tf-pipeline-state-lock-table"
      KeySchema:
        - AttributeName: "LockID"
          KeyType: "HASH"
      AttributeDefinitions:
        - AttributeName: "LockID"
          AttributeType: "S"
      BillingMode: "PAY_PER_REQUEST"
      PointInTimeRecoverySpecification:
        PointInTimeRecoveryEnabled: true
      SSESpecification:
        KMSMasterKeyId: !GetAtt KeyDynamoDB.Arn
        SSEEnabled: true
        SSEType: "KMS"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-state-lock-table"

#################################
# CloudWatch Logs (CodeBuild terraform plan)
#################################
  CWLTfplan:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfplan-project"
      RetentionInDays: !Ref BuildLogsRationDay
      KmsKeyId: !GetAtt KeyCWL.Arn
      Tags: 
        - Key: "Name"
          Value: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfplan-project"

#################################
# CodeBuild (terraform plan)
#################################
  PolicyTfplan:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${PrjPrefix}-tf-build-tfplan-project-policy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: "BuildLogs"
            Effect: "Allow"
            Action:
              - "logs:CreateLogGroup"
              - "logs:CreateLogStream"
              - "logs:PutLogEvents"
            Resource:
              - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CWLTfplan}"
              - !GetAtt CWLTfplan.Arn
          - Sid: "KmsKey"
            Effect: "Allow"
            Action:
              - "kms:Decrypt"
              - "kms:Encrypt"
              - "kms:ReEncrypt"
              - "kms:GenerateDataKey"
            Resource:
              - !GetAtt KeyS3Arthifact.Arn
              - !GetAtt KeyS3Tfstate.Arn
              - !GetAtt KeyDynamoDB.Arn
          - Sid: "S3Tfstate"
            Effect: "Allow"
            Action:
              - "s3:PutObject*"
              - "s3:DeleteObject*"
            Resource:
              - !Sub
                - "${BucketArn}/*"
                - { BucketArn: !GetAtt BucketTfstate.Arn }
          - Sid: "DynamoDB"
            Effect: "Allow"
            Action:
              - "dynamodb:PutItem"
              - "dynamodb:DeleteItem"
            Resource: !GetAtt DDBTable.Arn

  RoleTfplan:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-build-tfplan-project-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "codebuild.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - !Ref PolicyTfplan
        - "arn:aws:iam::aws:policy/ReadOnlyAccess"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfplan-project-role"

  ProjectTfplan:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Sub "${PrjPrefix}-tf-build-tfplan-project"
      Description: "Execute terraform plan command."
      Source:
        Type: "CODEPIPELINE"
        BuildSpec: |
          version: 0.2
          env:
            exported-variables:
              - BuildID
              - BuildTag
          phases:
            install:
              runtime-versions:
                golang: 1.14
              commands:
                - "git clone https://github.com/tfutils/tfenv.git ~/.tfenv"
                - "ln -s ~/.tfenv/bin/* /usr/local/bin"
                - "tfenv install $TF_VERSION"
                - "tfenv use $TF_VERSION"
            pre_build:
              commands:
                - "terraform init -input=false -no-color"
            build:
              commands:
                - "terraform plan -input=false -no-color"
            post_build:
              commands:
                - "export BuildID=`echo $CODEBUILD_BUILD_ID | cut -d: -f1`"
                - "export BuildTag=`echo $CODEBUILD_BUILD_ID | cut -d: -f2`"
      Artifacts:
        Type: "CODEPIPELINE"
      Cache:
        Type: "LOCAL"
        Modes:
          - "LOCAL_DOCKER_LAYER_CACHE"
      Environment:
        Type: "LINUX_CONTAINER"
        ComputeType: "BUILD_GENERAL1_SMALL"
        Image: "aws/codebuild/amazonlinux2-x86_64-standard:3.0"
        ImagePullCredentialsType: "CODEBUILD"
        PrivilegedMode: true
        EnvironmentVariables:
          - Name: "TF_VERSION"
            Type: PLAINTEXT
            Value: !Ref TerraformVersion
      LogsConfig:
        CloudWatchLogs:
          Status: "ENABLED"
          GroupName: !Ref CWLTfplan
      EncryptionKey: !GetAtt KeyS3Arthifact.Arn
      ResourceAccessRole: !GetAtt RoleTfplan.Arn
      ServiceRole: !GetAtt RoleTfplan.Arn
      TimeoutInMinutes: 60
      Visibility: "PRIVATE"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfplan-project"

#################################
# CloudWatch Logs (CodeBuild terraform apply)
#################################
  CWLTfapply:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfapply-project-role"
      RetentionInDays: !Ref BuildLogsRationDay
      KmsKeyId: !GetAtt KeyCWL.Arn
      Tags: 
        - Key: "Name"
          Value: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfapply-project-role"

#################################
# CodeBuild (terraform apply)
#################################
  RoleTfapply:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-build-tfapply-project-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "codebuild.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/AdministratorAccess"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfapply-project-role"

  ProjectTfapply:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Sub "${PrjPrefix}-tf-build-tfapply-project"
      Description: "Execute terraform apply command."
      Source:
        Type: "CODEPIPELINE"
        BuildSpec: |
          version: 0.2
          phases:
            install:
              runtime-versions:
                golang: 1.14
              commands:
                - "git clone https://github.com/tfutils/tfenv.git ~/.tfenv"
                - "ln -s ~/.tfenv/bin/* /usr/local/bin"
                - "tfenv install $TF_VERSION"
                - "tfenv use $TF_VERSION"
            pre_build:
              commands:
                - "terraform init -input=false -no-color"
            build:
              commands:
                - "terraform apply -input=false -no-color -auto-approve"
      Artifacts:
        Type: "CODEPIPELINE"
      Cache:
        Type: "LOCAL"
        Modes:
          - "LOCAL_DOCKER_LAYER_CACHE"
      Environment:
        Type: "LINUX_CONTAINER"
        ComputeType: "BUILD_GENERAL1_SMALL"
        Image: "aws/codebuild/amazonlinux2-x86_64-standard:3.0"
        ImagePullCredentialsType: "CODEBUILD"
        PrivilegedMode: true
        EnvironmentVariables:
          - Name: "TF_VERSION"
            Type: PLAINTEXT
            Value: !Ref TerraformVersion
      LogsConfig:
        CloudWatchLogs:
          Status: "ENABLED"
          GroupName: !Ref CWLTfapply
      EncryptionKey: !GetAtt KeyS3Arthifact.Arn
      ResourceAccessRole: !GetAtt RoleTfapply.Arn
      ServiceRole: !GetAtt RoleTfapply.Arn
      TimeoutInMinutes: 60
      Visibility: "PRIVATE"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfapply-project"

#################################
# CodePipeline
#################################
  PolicyTfPipeline:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${PrjPrefix}-tf-pipeline-policy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: "S3Artifact"
            Effect: "Allow"
            Action:
              - "s3:GetObject*"
              - "s3:GetBucket*"
              - "s3:List*"
              - "s3:DeleteObject*"
              - "s3:PutObject"
              - "s3:Abort*"
            Resource:
              - !GetAtt BucketArtifacts.Arn
              - !Sub
                - "${BucketArn}/*"
                - { BucketArn: !GetAtt BucketArtifacts.Arn }
          - Sid: "KmsKey"
            Effect: "Allow"
            Action:
              - "kms:Decrypt"
              - "kms:DescribeKey"
              - "kms:Encrypt"
              - "kms:ReEncrypt*"
              - "kms:GenerateDataKey*"
            Resource:
              - !GetAtt KeyS3Arthifact.Arn
          - Sid: "CodeCommitRepo"
            Effect: "Allow"
            Action:
              - "codecommit:GetBranch"
              - "codecommit:GetCommit"
              - "codecommit:UploadArchive"
              - "codecommit:GetUploadArchiveStatus"
              - "codecommit:CancelUploadArchive"
            Resource:
              - !GetAtt CodeCommit.Arn
          - Sid: "CodeBuildProjects"
            Effect: "Allow"
            Action:
              - "codebuild:BatchGetBuilds"
              - "codebuild:StartBuild"
              - "codebuild:StopBuild"
            Resource:
              - !GetAtt ProjectTfsec.Arn
              - !GetAtt ProjectTfplan.Arn
              - !GetAtt ProjectTfapply.Arn

  RoleTfPipelne:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-pipeline-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "codepipeline.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - !Ref PolicyTfPipeline
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-role"

  CodePipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      Name: !Sub "${PrjPrefix}-tf-pipeline"
      ArtifactStore:
        EncryptionKey:
          Id: !GetAtt KeyS3Arthifact.Arn
          Type: "KMS"
        Location: !Ref BucketArtifacts
        Type: "S3"
      RoleArn: !GetAtt RoleTfPipelne.Arn
      Stages:
        - Name: "Source"
          Actions:
            - Name: "CodeCommit_Source"
              ActionTypeId:
                Category: "Source"
                Owner: "AWS"
                Provider: "CodeCommit"
                Version: "1"
              Configuration:
                RepositoryName: !GetAtt CodeCommit.Name
                BranchName: !Ref BranchName
                PollForSourceChanges: false
              OutputArtifacts:
                - Name: "Artifact_Source_CodeCommit_Source"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 1
        - Name: "tfsec_Stage"
          Actions:
            - Name: "Terraform_Security_Analysis"
              Namespace: TFSEC
              ActionTypeId:
                Category: "Build"
                Owner: "AWS"
                Provider: "CodeBuild"
                Version: "1"
              Configuration:
                ProjectName: !Ref ProjectTfsec
              InputArtifacts:
                - Name: "Artifact_Source_CodeCommit_Source"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 1
        - Name: "Terraform_Stages"
          Actions:
            - Name: "Terraform_Security_Analysis_Manual_Review"
              ActionTypeId:
                Category: "Approval"
                Owner: "AWS"
                Provider: "Manual"
                Version: "1"
              Configuration:
                CustomData: "tfsec review"
                ExternalEntityLink: !Sub "https://${AWS::Region}.console.aws.amazon.com/codesuite/codebuild/${AWS::AccountId}/projects/#{TFSEC.BuildID}/build/#{TFSEC.BuildID}%3A#{TFSEC.BuildTag}/?region=${AWS::Region}"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 1
            - Name: "Terraform_Plan"
              Namespace: "TERRAFORM"
              ActionTypeId:
                Category: "Build"
                Owner: "AWS"
                Provider: "CodeBuild"
                Version: "1"
              Configuration:
                ProjectName: !Ref ProjectTfplan
              InputArtifacts:
                - Name: "Artifact_Source_CodeCommit_Source"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 2
            - Name: "Terraform_Plan_Manual_Review"
              ActionTypeId:
                Category: "Approval"
                Owner: "AWS"
                Provider: "Manual"
                Version: "1"
              Configuration:
                CustomData: "Terraform plan review"
                ExternalEntityLink: !Sub "https://${AWS::Region}.console.aws.amazon.com/codesuite/codebuild/${AWS::AccountId}/projects/#{TERRAFORM.BuildID}/build/#{TERRAFORM.BuildID}%3A#{TERRAFORM.BuildTag}/?region=${AWS::Region}"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 3
            - Name: "Terraform_Apply"
              ActionTypeId:
                Category: "Build"
                Owner: "AWS"
                Provider: "CodeBuild"
                Version: "1"
              Configuration:
                ProjectName: !Ref ProjectTfapply
              InputArtifacts:
                - Name: "Artifact_Source_CodeCommit_Source"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 4

#################################
# EventBridge (CodeCommit State Change)
# #################################
  PolicyEventBridge:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${PrjPrefix}-tf-pipeline-event-policy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: "CodePipelineExec"
            Effect: "Allow"
            Action: "codepipeline:StartPipelineExecution"
            Resource: !Sub "arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:${CodePipeline}"
  
  RoleEventBridge:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-pipeline-event-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "events.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - !Ref PolicyEventBridge
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-event-role"
  
  EventsTfPipeline:
    Type: AWS::Events::Rule
    Properties:
      Name: !Sub "${PrjPrefix}-tf-pipeline-event"
      EventPattern:
        source:
          - "aws.codecommit"
        resources:
          - !GetAtt CodeCommit.Arn
        detail-type:
          - "CodeCommit Repository State Change"
        detail:
          event:
            - referenceCreated
            - referenceUpdated
          referenceType:
            - "branch"
          referenceName:
            - !Ref BranchName
      State: "ENABLED"
      Targets:
        - Arn: !Sub "arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:${CodePipeline}"
          Id: "TerraformPipeline"
          RoleArn: !GetAtt RoleEventBridge.Arn

Outputs:
  TfstateBucketName:
    Value: !Ref BucketTfstate
    Export:
      Name: !Sub "${AWS::StackName}-TfstateBucketName"
  DDBTableName:
    Value: !Ref DDBTable
    Export:
      Name: !Sub "${AWS::StackName}-DDBTableName"

CodeCommit、CodeBuild(tfsec)の実装

まずは、以下の赤線の部分を実装します。

KMS

各種リソースの暗号化を行うためにKMSキーを作成します。

今回、暗号化するリソースは以下の通りです。

  • CloudWatch Logs(ビルドログ、Lambdaログ)
  • S3(アーティファクト、tfstate用バケット)
  • Secrets Manager(Dockerのログイン情報)
  • DynamoDB(terraformコマンドの排他制御)

今回は、シンプルにIAMリソースに権限を寄せる方向でいこうと思います。

実装部分

Resources:
#################################
# KMS (CloudWatch Logs)
#################################
  KeyCWL:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline Build Logs Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
          - Sid: Allow use of the key from CWL
            Effect: "Allow"
            Principal:
              Service: !Sub "logs.${AWS::Region}.amazonaws.com"
            Action:
              - "kms:Encrypt"
              - "kms:Decrypt"
              - "kms:ReEncrypt"
              - "kms:GenerateDataKey"
              - "kms:Describe"
            Resource: "*"
            Condition:
              ArnLike:
                kms:EncryptionContext:aws:logs:arn: !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-logs-key"

  KeyAliasCWL:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-logs-key"
      TargetKeyId: !Ref KeyCWL

#################################
# KMS (S3 arthifact)
#################################
  KeyS3Arthifact:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline arthifact Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-artifact-key"

  KeyAliasS3Arthifact:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-artifact-key"
      TargetKeyId: !Ref KeyS3Arthifact

#################################
# KMS (S3 tfstate)
#################################
  KeyS3Tfstate:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline tfstate Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-tfstate-key"

  KeyAliasS3Tfstate:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-tfstate-key"
      TargetKeyId: !Ref KeyS3Tfstate

#################################
# KMS (Secrets Manager)
#################################
  KeySecretsManager:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline Secrets Manager Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-secretsmanager-key"

  KeyAliasSecretsManager:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-secretsmanager-key"
      TargetKeyId: !Ref KeySecretsManager

#################################
# KMS (DynamoDB)
#################################
  KeyDynamoDB:
    Type: AWS::KMS::Key
    Properties:
      Description: "Terraform pipeline DynamoDB Key"
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: "Enable IAM User Permissions"
            Effect: "Allow"
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
      KeySpec: "SYMMETRIC_DEFAULT"
      KeyUsage: "ENCRYPT_DECRYPT"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-dynamoDB-key"

  KeyAliasDynamoDB:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub "alias/${PrjPrefix}-tf-pipeline-dynamoDB-key"
      TargetKeyId: !Ref KeyDynamoDB

カイゼンできる部分

上記のキーポリシーでは、「スタックを作成するアカウントのルートユーザー」と、「KMSの操作権限を持っているIAMリソース」にキーの操作を許可するように定義されています。

もし、大きな権限を持ったIAMリソースが意図せず作成された場合、今回の鍵で暗号化したリソースは復号されてしまう恐れがあります。そのため、本番運用する場合はキーポリシーも適切に権限管理する必要があります。

キーポリシーも厳重に管理したい方は、以下を重ねてご覧ください。

CloudWatch Logs

S3

Secrets Manager

DynamoDB

CodeCommit

続いて、Terraformのコードを格納するCodeCommitレポジトリを作成します。

実装部分

Resources:
#################################
# CodeCommit
#################################
  CodeCommit:
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName: !Sub "${PrjPrefix}-tf-repo"
      RepositoryDescription: !Sub "${PrjPrefix}-tf-repo"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-repo"

リポジトリ作成時にファイルを作成する

今回、「exclude.yml」というファイルを、「Lambda-backed カスタムリソース」で自動的に作成します。

exclude.yml」は、tfsecで検知されたルールを無視(ignore)する際に使用します。

カスタムリソースの詳しい実装方法は、以下ブログをご覧ください。

実装部分

Resources:
#################################
# Custom Resource
#################################
  PutExcludeRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        - "arn:aws:iam::aws:policy/AWSCodeCommitPowerUser"
      RoleName: !Sub "${PrjPrefix}-tf-repo-${BranchName}-put-exclude-role"

  PutExcludeLog:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/lambda/${PrjPrefix}-tf-repo-${BranchName}-put-exclude"
      KmsKeyId: !GetAtt KeyCWL.Arn

  PutExcludeFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Sub "${PrjPrefix}-tf-repo-${BranchName}-put-exclude"
      Code:
        ZipFile: |
          import json
          import boto3
          import cfnresponse

          def handler(event, context):
              try:
                  repository = event['ResourceProperties']['RepositoryName']
                  branch = event['ResourceProperties']['BranchName']
                  content = event['ResourceProperties']['FileContent'].encode()
                  path = event['ResourceProperties']['FilePath']

                  if event['RequestType'] == 'Create':
                      codecommit = boto3.client('codecommit')
                      response = codecommit.put_file(
                          repositoryName=repository,
                          branchName=branch,
                          fileContent=content,
                          filePath=path,
                          commitMessage='Initial Commit',
                          name='Your Lambda Helper'
                      )
                      cfnresponse.send(event, context, cfnresponse.SUCCESS, response)
                  if event['RequestType'] == 'Delete':
                      cfnresponse.send(event, context, cfnresponse.SUCCESS, {'Response': 'Success'})
                  if event['RequestType'] == 'Update':
                      cfnresponse.send(event, context, cfnresponse.SUCCESS, {'Response': 'Success'})
              except Exception as e:
                  print(e)
                  cfnresponse.send(event, context, cfnresponse.FAILED, {})
      Handler: index.handler
      MemorySize: 128
      Role: !GetAtt PutExcludeRole.Arn
      Runtime: "python3.9"
      Timeout: 60
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-repo-${BranchName}-put-exclude"
    DependsOn: PutExcludeLog

  PutExclude:
    Type: Custom::CodeCommitPutExclude
    Properties:
      ServiceToken: !GetAtt PutExcludeFunction.Arn
      RepositoryName: !GetAtt CodeCommit.Name
      BranchName: !Ref BranchName
      FileContent: "---\nexclude:"
      FilePath: "exclude.yml"

Secrets Manager

tfsecを実行するCodeBuildでは、Aqua Security(tfsecの開発元)のDockerイメージを使用します。

Docker Hubからイメージを取得するため、「レート制限」に引っかからないよう、Docker Hubへの認証情報をSecrets Managerに保存します。

Docker HubでTwo-Factor Authentication(MFA)を有効にしている場合は、passwordの代わりに「AccessToken」をDocker Hubで発行し設定します。

CodeBuildを扱う上でのDocker Hubのレート制限」についてもっと知りたい方は、以下も重ねてご覧ください。

実装部分

Resources:
#################################
# Secrets Manager
#################################
  DockerLoginProfile:
    Type: AWS::SecretsManager::Secret
    Properties:
      Description: "Docker login profile for terraform pipelines"
      Name: !Sub "${PrjPrefix}/docker_hub"
      SecretString: !Sub '{"username":"${DockerHubUserName}","password":"${DockerHubUserPassword}"}'
      KmsKeyId: !GetAtt KeySecretsManager.Arn
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}/docker_hub"

CloudWatch Logs(CodeBuild:tfsec)

tfsec用のCodeBuildのビルドログを保管するCloudWatch Logsを設定します。

保持期間はParametersセクションから変更できるようにします。

実装部分

Parameters:
  BuildLogsRationDay:
  Type: Number
  Description: "Enter build log retention period."
  Default: 90
  AllowedValues:
    - 1
    - 3
    - 5
    - 7
    - 14
    - 30
    - 60
    - 90
    - 120
    - 150
    - 180
    - 365
    - 400
    - 545
    - 731
    - 1827
    - 3653

Resources:
#################################
# CloudWatch Logs (CodeBuild tfsec)
#################################
  CWLTfsec:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfsec-project"
      RetentionInDays: !Ref BuildLogsRationDay
      KmsKeyId: !GetAtt KeyCWL.Arn
      Tags: 
        - Key: "Name"
          Value: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfsec-project"

S3(Artifacts)

CodePipelineで使用するアーティファクト用のS3バケットを作成します。

CodeBuildのIAMロールでResourceセクションを厳密に制御するため、ビルドプロジェクトより先に作成します。

暗号化キーは先ほど作成した鍵を使用してSSE-KMSとします。「バケットキー」を使用してリクエストコストの削減も行います。

また、オブジェクトACLは機能的に問題ないため、「無効」で設定します。

実装部分

#################################
# S3 Bucket (Artifact)
#################################
  BucketArtifacts:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub "${PrjPrefix}-tf-pipeline-artifacts"
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: "aws:kms"
              KMSMasterKeyID: !GetAtt KeyS3Arthifact.Arn
            BucketKeyEnabled: true
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-artifacts"

  BucketPolicyArtifacts:
      Type: AWS::S3::BucketPolicy
      Properties: 
        Bucket: !Ref BucketArtifacts
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Sid: "AllowSSLRequestsOnly"
              Effect: "Deny"
              Principal: "*"
              Action: "s3:*"
              Resource:
                - !GetAtt BucketArtifacts.Arn
                - !Sub
                  - "${BucketArn}/*"
                  - { BucketArn: !GetAtt BucketArtifacts.Arn }
              Condition:
                Bool:
                  aws:SecureTransport: "false"

カイゼンできる部分

Security Hubの「AWS Foundational Security Best Practices standard」に従い、セキュリティレベルを高めることができます。

具体的には、以下のコントロールが今回の実装では対応できていません。必要に応じて設定を行いましょう。

  • [S3.9]S3バケットサーバーアクセスログ記録を有効にする必要があります
  • [S3.10]バージョニングが有効なS3バケットでは、ライフサイクルポリシーを設定する必要があります
  • [S3.11]S3バケットでは、イベント通知を有効にする必要があります

CodeBuild(tfsec)

tfsecが搭載されたDockerイメージで、ビルドプロジェクトを作成します。

Secrets Managerに保管したDocker Hubの認証情報も起動時に使用するよう作成します。

実装部分

#################################
# CodeBuild (tfsec)
#################################
  PolicyTfsec:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${PrjPrefix}-tf-build-tfsec-project-policy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: "BuildLogs"
            Effect: "Allow"
            Action:
              - "logs:CreateLogGroup"
              - "logs:CreateLogStream"
              - "logs:PutLogEvents"
            Resource:
              - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CWLTfsec}"
              - !GetAtt CWLTfsec.Arn
          - Sid: "BuildReports"
            Effect: "Allow"
            Action:
              - "codebuild:CreateReportGroup"
              - "codebuild:CreateReport"
              - "codebuild:UpdateReport"
              - "codebuild:BatchPutTestCases"
              - "codebuild:BatchPutCodeCoverages"
            Resource:
              - !Sub "arn:${AWS::Partition}:codebuild:${AWS::Region}:${AWS::AccountId}:report-group/${PrjPrefix}-tf-build-tfsec-project-reports"
          - Sid: "GitPull"
            Effect: "Allow"
            Action: "codecommit:GitPull"
            Resource: !GetAtt CodeCommit.Arn
          - Sid: "GetSecretValue"
            Effect: "Allow"
            Action: "secretsmanager:GetSecretValue"
            Resource: !Ref DockerLoginProfile
          - Sid: "KmsKey"
            Effect: "Allow"
            Action:
              - "kms:Decrypt"
              - "kms:DescribeKey"
              - "kms:Encrypt"
              - "kms:ReEncrypt"
              - "kms:GenerateDataKey"
            Resource:
              - !GetAtt KeyS3Arthifact.Arn
              - !GetAtt KeySecretsManager.Arn
          - Sid: "S3Artifact"
            Effect: "Allow"
            Action:
              - "s3:PutObject"
              - "s3:GetObject"
              - "s3:GetObjectVersion"
              - "s3:GetBucketAcl"
              - "s3:GetBucketLocation"
            Resource:
              - !GetAtt BucketArtifacts.Arn
              - !Sub
                - "${BucketArn}/*"
                - { BucketArn: !GetAtt BucketArtifacts.Arn }

  RoleTfsec:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-build-tfsec-project-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "codebuild.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - !Ref PolicyTfsec
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfsec-project-role"

  ProjectTfsec:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Sub "${PrjPrefix}-tf-build-tfsec-project"
      Description: "Analyze the code for vulnerabilities using tfsec."
      Source:
        Type: "CODEPIPELINE"
        BuildSpec: |
          version: 0.2
          env:
            exported-variables:
              - BuildID
              - BuildTag
          phases:
            pre_build:
              commands:
                - "echo Executing tfsec"
                - "mkdir -p reports/tfsec/"
            build:
              commands:
                - "tfsec -s --no-color --config-file exclude.yml ."
                - "tfsec -s --no-color --config-file exclude.yml . --format junit > reports/tfsec/report.xml"
            post_build:
              commands:
                - "export BuildID=`echo $CODEBUILD_BUILD_ID | cut -d: -f1`"
                - "export BuildTag=`echo $CODEBUILD_BUILD_ID | cut -d: -f2`"
          reports:
            reports:
              files:
                - "reports/tfsec/report.xml"
              file-format: JUNITXML
      Artifacts:
        Type: "CODEPIPELINE"
      Cache:
        Type: "LOCAL"
        Modes:
          - "LOCAL_DOCKER_LAYER_CACHE"
      Environment:
        Type: "LINUX_CONTAINER"
        ComputeType: "BUILD_GENERAL1_SMALL"
        Image: "aquasec/tfsec:latest"
        ImagePullCredentialsType: "SERVICE_ROLE"
        RegistryCredential:
          Credential: !Ref DockerLoginProfile
          CredentialProvider: "SECRETS_MANAGER"
        PrivilegedMode: true
      LogsConfig:
        CloudWatchLogs:
          Status: "ENABLED"
          GroupName: !Ref CWLTfsec
      EncryptionKey: !GetAtt KeyS3Arthifact.Arn
      ResourceAccessRole: !GetAtt RoleTfsec.Arn
      ServiceRole: !GetAtt RoleTfsec.Arn
      TimeoutInMinutes: 60
      Visibility: "PRIVATE"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfsec-project"

  ReportGroupTfsec:
    Type: AWS::CodeBuild::ReportGroup
    Properties:
      Name: !Sub "${ProjectTfsec}-reports"
      Type: "TEST"
      DeleteReports: true
      ExportConfig:
        ExportConfigType: "NO_EXPORT"
      Tags: 
        - Key: "Name"
          Value: !Sub "${ProjectTfsec}-reports"

buildspec.ymlの解説

ここでは、ビルドプロジェクトに設定したbuildspec.ymlについて解説します。

buildspec.yml

  version: 0.2
  env:
    exported-variables:
      - BuildID
      - BuildTag
  phases:
    pre_build:
      commands:
        - "echo Executing tfsec"
        - "mkdir -p reports/tfsec/"
    build:
      commands:
        - "tfsec -s --no-color --config-file exclude.yml ."
        - "tfsec -s --no-color --config-file exclude.yml . --format junit > reports/tfsec/report.xml"
    post_build:
      commands:
        - "export BuildID=`echo $CODEBUILD_BUILD_ID | cut -d: -f1`"
        - "export BuildTag=`echo $CODEBUILD_BUILD_ID | cut -d: -f2`"
  reports:
    reports:
      files:
        - "reports/tfsec/report.xml"
      file-format: JUNITXML

env/exported-variablesでは、ビルドプロジェクトで使用した値をエクスポートするために使用します。

「BuildID」と「BuildTag」をエクスポートするように設定しています。

各エクスポートした値は、CodePipelineの手動承認アクションで使用します。

buildフェーズで、tfsecを実行しています。

--no-colorオプションを使用することで、CodeBuildのログの文字化けを防ぎます。私も実装当初、以下のような事象に遭遇しエラー解決まで時間がかかりました。

--config-fileオプションでは、指定したファイルから無視する検知ルールを設定します。

CodeCommit作成時に、カスタムリソースで作成したexclude.ymlが、--config-fileの対象になっています。

--config-fileオプションで指定するファイルは空ファイルでも問題なく動作します。

-format junitでレポートを「JUnit XML形式」で出力しています。

CodeBuildレポートグループでは、「JUnit XML形式」をサポートしています。

Backend(S3, DynamoDB)の実装

ここからは、以下の赤線の部分を実装します。

S3

tfstate用のS3バケットを作成します。

アーティファクト用と違い、tfstate用のS3バケットでは、バージョニング機能を設定します。

実装部分

#################################
# S3 Bucket (tfstate)
#################################
  BucketTfstate:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub "${PrjPrefix}-tf-pipeline-tfstate"
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: "aws:kms"
              KMSMasterKeyID: !GetAtt KeyS3Tfstate.Arn
            BucketKeyEnabled: true
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: "Enabled"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-tfstate"

  BucketPolicyTfstate:
      Type: AWS::S3::BucketPolicy
      Properties: 
        Bucket: !Ref BucketTfstate
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Sid: "AllowSSLRequestsOnly"
              Effect: "Deny"
              Principal: "*"
              Action: "s3:*"
              Resource:
                - !GetAtt BucketTfstate.Arn
                - !Sub
                  - "${BucketArn}/*"
                  - { BucketArn: !GetAtt BucketTfstate.Arn }
              Condition:
                Bool:
                  aws:SecureTransport: "false"

DynamoDB

terraformコマンドの排他制御を実装するため、DynamoDBテーブルを作成します。主キーには、LockIDを設定します。

実装部分

#################################
# DynamoDB table (state lock)
#################################
  DDBTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Sub "${PrjPrefix}-tf-pipeline-state-lock-table"
      KeySchema:
        - AttributeName: "LockID"
          KeyType: "HASH"
      AttributeDefinitions:
        - AttributeName: "LockID"
          AttributeType: "S"
      BillingMode: "PAY_PER_REQUEST"
      PointInTimeRecoverySpecification:
        PointInTimeRecoveryEnabled: true
      SSESpecification:
        KMSMasterKeyId: !GetAtt KeyDynamoDB.Arn
        SSEEnabled: true
        SSEType: "KMS"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-state-lock-table"

CodeBuild(plan)、CodeBuild(apply)の実装

ここからは、以下の赤線の部分を実装します。

tfsecを行うCodeBuildと違い、ECRからAmazon LinuxのDockerイメージを使用します。

CodeBuild(plan)

terraform plan用のCodeBuildプロジェクトでは、以下のブログによると「ReadOnlyAccess」と「DynamoDBへの操作権限」が必要です。

加えて、今回は以下のポリシーが必要です。

  • アーティファクト用のS3バケットへの操作権限
  • S3、DynamoDBの暗号化で使用するKMSキーへの操作権限
  • CloudWatch Logsへのビルドログの配信権限

実装部分

#################################
# CloudWatch Logs (CodeBuild terraform plan)
#################################
  CWLTfplan:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfplan-project"
      RetentionInDays: !Ref BuildLogsRationDay
      KmsKeyId: !GetAtt KeyCWL.Arn
      Tags: 
        - Key: "Name"
          Value: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfplan-project"

#################################
# CodeBuild (terraform plan)
#################################
  PolicyTfplan:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${PrjPrefix}-tf-build-tfplan-project-policy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: "BuildLogs"
            Effect: "Allow"
            Action:
              - "logs:CreateLogGroup"
              - "logs:CreateLogStream"
              - "logs:PutLogEvents"
            Resource:
              - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CWLTfplan}"
              - !GetAtt CWLTfplan.Arn
          - Sid: "KmsKey"
            Effect: "Allow"
            Action:
              - "kms:Decrypt"
              - "kms:Encrypt"
              - "kms:ReEncrypt"
              - "kms:GenerateDataKey"
            Resource:
              - !GetAtt KeyS3Arthifact.Arn
              - !GetAtt KeyS3Tfstate.Arn
              - !GetAtt KeyDynamoDB.Arn
          - Sid: "S3Tfstate"
            Effect: "Allow"
            Action:
              - "s3:PutObject*"
              - "s3:DeleteObject*"
            Resource:
              - !Sub
                - "${BucketArn}/*"
                - { BucketArn: !GetAtt BucketTfstate.Arn }
          - Sid: "DynamoDB"
            Effect: "Allow"
            Action:
              - "dynamodb:PutItem"
              - "dynamodb:DeleteItem"
            Resource: !GetAtt DDBTable.Arn

  RoleTfplan:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-build-tfplan-project-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "codebuild.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - !Ref PolicyTfplan
        - "arn:aws:iam::aws:policy/ReadOnlyAccess"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfplan-project-role"

  ProjectTfplan:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Sub "${PrjPrefix}-tf-build-tfplan-project"
      Description: "Execute terraform plan command."
      Source:
        Type: "CODEPIPELINE"
        BuildSpec: |
          version: 0.2
          env:
            exported-variables:
              - BuildID
              - BuildTag
          phases:
            install:
              runtime-versions:
                golang: 1.14
              commands:
                - "git clone https://github.com/tfutils/tfenv.git ~/.tfenv"
                - "ln -s ~/.tfenv/bin/* /usr/local/bin"
                - "tfenv install $TF_VERSION"
                - "tfenv use $TF_VERSION"
            pre_build:
              commands:
                - "terraform init -input=false -no-color"
            build:
              commands:
                - "terraform plan -input=false -no-color"
            post_build:
              commands:
                - "export BuildID=`echo $CODEBUILD_BUILD_ID | cut -d: -f1`"
                - "export BuildTag=`echo $CODEBUILD_BUILD_ID | cut -d: -f2`"
      Artifacts:
        Type: "CODEPIPELINE"
      Cache:
        Type: "LOCAL"
        Modes:
          - "LOCAL_DOCKER_LAYER_CACHE"
      Environment:
        Type: "LINUX_CONTAINER"
        ComputeType: "BUILD_GENERAL1_SMALL"
        Image: "aws/codebuild/amazonlinux2-x86_64-standard:3.0"
        ImagePullCredentialsType: "CODEBUILD"
        PrivilegedMode: true
        EnvironmentVariables:
          - Name: "TF_VERSION"
            Type: PLAINTEXT
            Value: !Ref TerraformVersion
      LogsConfig:
        CloudWatchLogs:
          Status: "ENABLED"
          GroupName: !Ref CWLTfplan
      EncryptionKey: !GetAtt KeyS3Arthifact.Arn
      ResourceAccessRole: !GetAtt RoleTfplan.Arn
      ServiceRole: !GetAtt RoleTfplan.Arn
      TimeoutInMinutes: 60
      Visibility: "PRIVATE"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfplan-project"

buildspec.ymlの解説

terraform plan用のビルドプロジェクトでも、簡単ではありますがbuildspec.ymlについて解説します。

buildspec.yml

  version: 0.2
  env:
    exported-variables:
      - BuildID
      - BuildTag
  phases:
    install:
      runtime-versions:
        golang: 1.14
      commands:
        - "git clone https://github.com/tfutils/tfenv.git ~/.tfenv"
        - "ln -s ~/.tfenv/bin/* /usr/local/bin"
        - "tfenv install $TF_VERSION"
        - "tfenv use $TF_VERSION"
    pre_build:
      commands:
        - "terraform init -input=false -no-color"
    build:
      commands:
        - "terraform plan -input=false -no-color"
    post_build:
      commands:
        - "export BuildID=`echo $CODEBUILD_BUILD_ID | cut -d: -f1`"
        - "export BuildTag=`echo $CODEBUILD_BUILD_ID | cut -d: -f2`"

env/exported-variablesはtfsec用のビルドプロジェクトと同様に、CodePipelineの手動承認アクションで使用するためエクスポートしています。

installフェーズでは、普段使っていることもあり、terraformのバージョン管理ツールとして「tfenv」をインストールしました。

CodeBuild(apply)

terraform apply用のビルドプロジェクトでは、「AdministratorAccess」権限をIAMロールに付与します。

かなり強めな権限のため、必要に応じて権限周りの制限を行いましょう。

terraform applyのビルドプロジェクトは、terraform plan用とそこまで大差がないためbuildspec.ymlの解説は省略します。

実装部分

#################################
# CloudWatch Logs (CodeBuild terraform apply)
#################################
  CWLTfapply:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfapply-project-role"
      RetentionInDays: !Ref BuildLogsRationDay
      KmsKeyId: !GetAtt KeyCWL.Arn
      Tags: 
        - Key: "Name"
          Value: !Sub "/aws/codebuild/${PrjPrefix}-tf-build-tfapply-project-role"

#################################
# CodeBuild (terraform apply)
#################################
  RoleTfapply:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-build-tfapply-project-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "codebuild.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/AdministratorAccess"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfapply-project-role"

  ProjectTfapply:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Sub "${PrjPrefix}-tf-build-tfapply-project"
      Description: "Execute terraform apply command."
      Source:
        Type: "CODEPIPELINE"
        BuildSpec: |
          version: 0.2
          phases:
            install:
              runtime-versions:
                golang: 1.14
              commands:
                - "git clone https://github.com/tfutils/tfenv.git ~/.tfenv"
                - "ln -s ~/.tfenv/bin/* /usr/local/bin"
                - "tfenv install $TF_VERSION"
                - "tfenv use $TF_VERSION"
            pre_build:
              commands:
                - "terraform init -input=false -no-color"
            build:
              commands:
                - "terraform apply -input=false -no-color -auto-approve"
      Artifacts:
        Type: "CODEPIPELINE"
      Cache:
        Type: "LOCAL"
        Modes:
          - "LOCAL_DOCKER_LAYER_CACHE"
      Environment:
        Type: "LINUX_CONTAINER"
        ComputeType: "BUILD_GENERAL1_SMALL"
        Image: "aws/codebuild/amazonlinux2-x86_64-standard:3.0"
        ImagePullCredentialsType: "CODEBUILD"
        PrivilegedMode: true
        EnvironmentVariables:
          - Name: "TF_VERSION"
            Type: PLAINTEXT
            Value: !Ref TerraformVersion
      LogsConfig:
        CloudWatchLogs:
          Status: "ENABLED"
          GroupName: !Ref CWLTfapply
      EncryptionKey: !GetAtt KeyS3Arthifact.Arn
      ResourceAccessRole: !GetAtt RoleTfapply.Arn
      ServiceRole: !GetAtt RoleTfapply.Arn
      TimeoutInMinutes: 60
      Visibility: "PRIVATE"
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-build-tfapply-project"

CodePipelineで結合

いよいよ最後のフェーズである「CodePipelineで結合」を行います。

今回は、パイプラインの要所で「手動承認」を組み込みます。

実装部分

#################################
# CodePipeline
#################################
  PolicyTfPipeline:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${PrjPrefix}-tf-pipeline-policy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: "S3Artifact"
            Effect: "Allow"
            Action:
              - "s3:GetObject*"
              - "s3:GetBucket*"
              - "s3:List*"
              - "s3:DeleteObject*"
              - "s3:PutObject"
              - "s3:Abort*"
            Resource:
              - !GetAtt BucketArtifacts.Arn
              - !Sub
                - "${BucketArn}/*"
                - { BucketArn: !GetAtt BucketArtifacts.Arn }
          - Sid: "KmsKey"
            Effect: "Allow"
            Action:
              - "kms:Decrypt"
              - "kms:DescribeKey"
              - "kms:Encrypt"
              - "kms:ReEncrypt*"
              - "kms:GenerateDataKey*"
            Resource:
              - !GetAtt KeyS3Arthifact.Arn
          - Sid: "CodeCommitRepo"
            Effect: "Allow"
            Action:
              - "codecommit:GetBranch"
              - "codecommit:GetCommit"
              - "codecommit:UploadArchive"
              - "codecommit:GetUploadArchiveStatus"
              - "codecommit:CancelUploadArchive"
            Resource:
              - !GetAtt CodeCommit.Arn
          - Sid: "CodeBuildProjects"
            Effect: "Allow"
            Action:
              - "codebuild:BatchGetBuilds"
              - "codebuild:StartBuild"
              - "codebuild:StopBuild"
            Resource:
              - !GetAtt ProjectTfsec.Arn
              - !GetAtt ProjectTfplan.Arn
              - !GetAtt ProjectTfapply.Arn

  RoleTfPipelne:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-pipeline-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "codepipeline.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - !Ref PolicyTfPipeline
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-role"

  CodePipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      Name: !Sub "${PrjPrefix}-tf-pipeline"
      ArtifactStore:
        EncryptionKey:
          Id: !GetAtt KeyS3Arthifact.Arn
          Type: "KMS"
        Location: !Ref BucketArtifacts
        Type: "S3"
      RoleArn: !GetAtt RoleTfPipelne.Arn
      Stages:
        - Name: "Source"
          Actions:
            - Name: "CodeCommit_Source"
              ActionTypeId:
                Category: "Source"
                Owner: "AWS"
                Provider: "CodeCommit"
                Version: "1"
              Configuration:
                RepositoryName: !GetAtt CodeCommit.Name
                BranchName: !Ref BranchName
                PollForSourceChanges: false
              OutputArtifacts:
                - Name: "Artifact_Source_CodeCommit_Source"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 1
        - Name: "tfsec_Stage"
          Actions:
            - Name: "Terraform_Security_Analysis"
              Namespace: TFSEC
              ActionTypeId:
                Category: "Build"
                Owner: "AWS"
                Provider: "CodeBuild"
                Version: "1"
              Configuration:
                ProjectName: !Ref ProjectTfsec
              InputArtifacts:
                - Name: "Artifact_Source_CodeCommit_Source"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 1
        - Name: "Terraform_Stages"
          Actions:
            - Name: "Terraform_Security_Analysis_Manual_Review"
              ActionTypeId:
                Category: "Approval"
                Owner: "AWS"
                Provider: "Manual"
                Version: "1"
              Configuration:
                CustomData: "tfsec review"
                ExternalEntityLink: !Sub "https://${AWS::Region}.console.aws.amazon.com/codesuite/codebuild/${AWS::AccountId}/projects/#{TFSEC.BuildID}/build/#{TFSEC.BuildID}%3A#{TFSEC.BuildTag}/?region=${AWS::Region}"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 1
            - Name: "Terraform_Plan"
              Namespace: "TERRAFORM"
              ActionTypeId:
                Category: "Build"
                Owner: "AWS"
                Provider: "CodeBuild"
                Version: "1"
              Configuration:
                ProjectName: !Ref ProjectTfplan
              InputArtifacts:
                - Name: "Artifact_Source_CodeCommit_Source"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 2
            - Name: "Terraform_Plan_Manual_Review"
              ActionTypeId:
                Category: "Approval"
                Owner: "AWS"
                Provider: "Manual"
                Version: "1"
              Configuration:
                CustomData: "Terraform plan review"
                ExternalEntityLink: !Sub "https://${AWS::Region}.console.aws.amazon.com/codesuite/codebuild/${AWS::AccountId}/projects/#{TERRAFORM.BuildID}/build/#{TERRAFORM.BuildID}%3A#{TERRAFORM.BuildTag}/?region=${AWS::Region}"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 3
            - Name: "Terraform_Apply"
              ActionTypeId:
                Category: "Build"
                Owner: "AWS"
                Provider: "CodeBuild"
                Version: "1"
              Configuration:
                ProjectName: !Ref ProjectTfapply
              InputArtifacts:
                - Name: "Artifact_Source_CodeCommit_Source"
              RoleArn: !GetAtt RoleTfPipelne.Arn
              RunOrder: 4

#################################
# EventBridge (CodeCommit State Change)
# #################################
  PolicyEventBridge:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${PrjPrefix}-tf-pipeline-event-policy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: "CodePipelineExec"
            Effect: "Allow"
            Action: "codepipeline:StartPipelineExecution"
            Resource: !Sub "arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:${CodePipeline}"
  
  RoleEventBridge:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${PrjPrefix}-tf-pipeline-event-role"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "events.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - !Ref PolicyEventBridge
      Tags:
        - Key: "Name"
          Value: !Sub "${PrjPrefix}-tf-pipeline-event-role"
  
  EventsTfPipeline:
    Type: AWS::Events::Rule
    Properties:
      Name: !Sub "${PrjPrefix}-tf-pipeline-event"
      EventPattern:
        source:
          - "aws.codecommit"
        resources:
          - !GetAtt CodeCommit.Arn
        detail-type:
          - "CodeCommit Repository State Change"
        detail:
          event:
            - referenceCreated
            - referenceUpdated
          referenceType:
            - "branch"
          referenceName:
            - !Ref BranchName
      State: "ENABLED"
      Targets:
        - Arn: !Sub "arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:${CodePipeline}"
          Id: "TerraformPipeline"
          RoleArn: !GetAtt RoleEventBridge.Arn

以上、アーキテクチャの解説でした。

いざ検証

ここからは、実際にCodeCommitにTerraformのコードをプッシュして検証してみます。

CloudFormationスタックは、以下のパラメーターで作成しました。

設定値 備考
BranchName main
BuildLogsRationDay 90
DockerHubUserName *** Docker Hubのユーザー名
DockerHubUserPassword *** Docker HubのAccessToken
PrjPrefix tks-prd 任意の値
TerraformVersion 1.2.0 任意の値

検知されるルールについて

今回、tfsecによって評価されるルールは全部で5つあります。

このうち「aws-iam-enforce-mfa」については、「exclude.yml」の動作検証のためにルールを無視するよう設定します。

CodeCommitへの接続

gitコマンドを使用してCodeComitへ接続します。認証情報を聞かれるためIAMコンソールから認証情報を事前に作成しておきましょう。

git clone https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/tks-prd-tf-repo

認証情報を入力します。

Username for 'https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/tks-prd-tf-repo': user
Password for 'https://user@git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/tks-prd-tf-repo': ******
remote: Counting objects: 3, done.
Unpacking objects: 100% (3/3), 231 bytes | 77.00 KiB/s, done.

terraformコードの作成

今回、検証用に以下のコードを用意しました。

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "4.19.0"
    }
  }
  backend "s3" {
    bucket = "tks-prd-tf-pipeline-tfstate" # S3(tfstate)で作成したバケット名
    key = "prd/terraform.tfstate" # tfstateを保管するための任意の格納先
    region = "ap-northeast-1"

    dynamodb_table = "tks-prd-tf-pipeline-state-lock-table" # 作成したDynamoDBテーブル
  }
}

provider "aws" {
  region = "ap-northeast-1"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  # source  = "../modules/vpc"
  version = "3.12.0"
  # insert the 23 required variables here
  cidr = "10.0.0.0/16"
  name = "terraform-reintroduction"
  public_subnets = ["10.0.0.0/24", "10.0.1.0/24"]
  azs = ["ap-northeast-1a", "ap-northeast-1c"]
}

resource "aws_security_group" "allow_tls" {
  name        = "allow_tls"
  description = "Allow TLS inbound traffic"
  vpc_id      = module.vpc.vpc_id

  ingress {
    description      = "TLS from VPC"
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks      = [module.vpc.vpc_cidr_block]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  tags = {
    Name = "allow_tls"
  }
}

data aws_caller_identity current {}
resource aws_iam_group developers {
  name =  "developersIO2022"
}

検知されるルールについて

今回、tfsecで評価されるルールは全部で5つあります。

このうち「aws-iam-enforce-mfa」については、「exclude.yml」の動作検証のためにルールを無視するよう設定します。

exclude.ymlの修正

まずは、exclude.ymlの中身を確認します。

exclude.yml

---
exclude:

exclude:の下に、配列で無視したいルールを記述します。

exclude.yml

  ---
  exclude:
+   - aws-iam-enforce-mfa

レポジトリへプッシュ

今回は、そのままレポジトリへプッシュします。

git add .
git commit -m "my first commit"
git push

tfsecの確認

CodeCommitレポジトリの状態変更が起こると、CodePipelineがトリガーされる仕様になっています。

画面を確認すると「手動承認」で止まっているため、ビルドログとレポートを確認してみます。

「レビュー」ボタンをクリックすると「レビュー用URL」が表示されるようにしています。

レビュー用URLは、CloudFormationで「https://${AWS::Region}.console.aws.amazon.com/codesuite/codebuild/${AWS::AccountId}/projects/#{TFSEC.BuildID}/build/#{TFSEC.BuildID}%3A#{TFSEC.BuildTag}/?region=${AWS::Region}」と動的に生成するよう定義しています。

buildspec.ymlでエクスポートした「BuildID」と「BuildTag」はこの実装のためにエクスポートしました。

生成したレポートURLからビルドログとレポートを確認してみます。

ビルドログを確認すると、tfsecでリポジトリのコードが評価されていることがわかります。

続いて「レポート」も確認してみます。

レポートでは、exclude.ymlで無視したルール以外の失敗したルールが一覧で表示されています。「レポートあたりのテストケースの最大数」がデフォルトでは500までのため、成功したテストケースについては表示していません。

成功したテストケースも表示したい場合は、tfsecコマンドに--include-passedオプションを入れると表示されます。

失敗したルールをクリックすると、「どのファイルのどの部分が失敗しているのか」を確認できます。

terraform plan/apply

今回はtfsecで検知したルールは是正せず、そのままパイプラインの完了まで行おうと思います。

「手動承認」を承諾して、「Terraform_Plan」アクションへ進めます。

ビルドログの抜粋になりますが、問題なくterraform planが走っていることがわかります。

terraform plan実行後も、「Terraform_Plan_Manual_Review」アクションで手動承認を組み込んでいます。tfsecの手動承認と同じ仕組みでレビュー用URLの作成をしています。

「承認をします」をクリックすると、terraform applyが実行されるため承認にはご注意ください。

最後に「Terraform_Apply」アクションで、terraform applyコマンドを実行します。

無事、リソースが作成されていることが確認できました。

参考

実装に当たってさまざまな人のTerraform Pipelineを見つけたためご紹介します。

おわりに

以上、「TerraformのCI/CDパイプラインを実装してみた」でした。今回、初めてCodePipelineを0から実装したため、かなり勉強になりました。

また、今回のような超大作ブログを長期休暇でかければいいなと思います。(比較的サーバーレスな構成のため課金に恐れずゆっくり書けました。)

このブログがどなたかのご参考になれば幸いです。

以上、AWS事業本部コンサルティング部のたかくにでした!