Transfer Family を使用して SFTP 経由でファイルを EFS に直接アップロードしてみた（VPCインターネット向け-鍵・パスワード認証編）

Transfer Family を使用して SFTP 経由でファイルを EFS に直接アップロードしてみました。 本記事ではID プロバイダーに「カスタムIDプロバイダー」、エンドポイントに「VPC（インターネット）」を選択した手順をご紹介します。

おつまみ

2022.12.22

こんにちは！AWS事業本部のおつまみです。

皆さん、Transfer Family を使用して SFTP 経由でファイルを EFS に直接アップロードしたいなぁと思ったことはありますか？私はあります。

Transfer Family は、AWS ストレージサービスとの間でファイルを送受信できる安全な転送サービスです。主にこれら4つの要素を選択し、構築します。

  • プロトコル
    • SFTP (SSH ファイル転送プロトコル) - Secure Shell 経由のファイル転送
    • AS2 (Applicability Statement 2) - ビジネス間データ交換用のメッセージングプロトコル情報
    • FTPS (File Transfer Protocol Secure) - TLS 暗号化を使用したファイル転送プロトコル
    • FTP (File Transfer Protocol) - 暗号化されていないファイル転送プロトコル
  • ID プロバイダー
    • サービスマネージド - サービス内でユーザーを作成および管理
    • AWS Directory Service情報 - AWS Managed AD でユーザーを有効にする、またはオンプレミス環境または AWS で独自のセルフマネージド AD を使用する
    • カスタム ID プロバイダー - 情報任意の ID プロバイダーとの統合によりユーザーを管理する
  • エンドポイント
    • パブリックアクセス - インターネット経由でアクセス可能
    • VPC（内部） - セキュリティグループを使用したアクセスコントロール可能、VPC内のリソースからのみアクセスする場合
    • VPC（インターネット） - セキュリティグループを使用したアクセスコントロール可能、VPC外からアクセスする場合
  • ドメイン
    • Amazon S3
    • Amazon EFS

ID プロバイダーにサービスマネージドを使用したいなぁと思った、そこのあなた！
サービスマネージドでは、鍵認証しか行えません。

ただシステム要件によっては、パスワード認証が必須な場合があると思います。
そこで今回はカスタム ID プロバイダーを使用し、鍵とパスワード両方の認証を有効にする方法をご紹介します。

そのため今回は以下を選択した構築方法の作業手順となります。

  • プロトコル
    • SFTP (SSH ファイル転送プロトコル) - Secure Shell 経由のファイル転送
  • ID プロバイダー
    • カスタム ID プロバイダー - 情報任意の ID プロバイダーとの統合によりユーザーを管理する
  • エンドポイント
    • VPC（インターネット） - セキュリティグループを使用したアクセスコントロール可能、VPC外からアクセスする場合
  • ドメイン
    • Amazon EFS

構成図

今回構築するアーキテクチャの構成図です。

ID管理はSecretsManagerのシークレットで設定します。
また不正にアクセスされないようセキュリティグループでIP制限を行います。

こちらのAWSにより公開されている方式を参考に構築しました。

作業手順

0.前提

下記ブログの「0. 事前準備 鍵の作成」~「4. EFSの作成」を参考に、以下のリソースをあらかじめ作成しておきます。

必要なリソース　　

  • 公開鍵・秘密鍵
  • VPC
  • EIP
  • セキュリティグループ
  • EFS
  • IAM

1. 公開されているSAMテンプレートの修正

先ほど紹介したAWSブログのthis packageを押下して、構築に必要なSAMテンプレートをダウンロードします。

ダウンロードされたテンプレートを今回の構築用に修正します。

1.1 SAMテンプレート（template.yml）の修正

変更箇所　　

  • Parameterセクション
    • TransferEndpointTypeをコメントアウト（26~33行目）
    • SecurityGroupIdsを追加（46~49行目下）
    • EIPを追加（50~53行目下）
    • TransferVPCEndpointをコメントアウト（72~75行目）
  • Resourceセクション
    • TransferServerをEFSに対応するように修正（102~124行目）
修正前 template.yml

template.yml

---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: A basic template for creating a Lambda-backed API Gateway for use as
  a custom identity provider in AWS Transfer. It authenticates against an
  entry in AWS Secrets Manager of the format server-id/username. Additionaly, the secret
  must hold the key-value pairs for all user properties returned to AWS Transfer.
  If you choose to also launch the AWS Transfer endpoint it will be provisioned with the SFTP protocol.
  This can be changed after launch. Note that FTP is only supported on VPC endpoints
  You can modify the Lambda function code to do something different after deployment.
Parameters:
  CreateServer:
    AllowedValues:
      - 'true'
      - 'false'
    Type: String
    Description: Whether this stack creates an AWS Transfer endpoint or not. If the endpoint is created as
      part of the stack, the custom identity provider is automatically associated with it.
    Default: 'true'
  SecretsManagerRegion:
    Type: String
    Description: (Optional) The region the secrets are stored in. If this value is not provided, the
      region this stack is deployed in will be used. Use this field if you are deploying this stack in
      a region where SecretsMangager is not available.
    Default: ''
  TransferEndpointType:
    AllowedValues:
      - 'PUBLIC'
      - 'VPC'
    Type: String
    Default: 'PUBLIC'
    Description: Select PUBLIC if you want a public facing AWS Transfer endpoint or VPC if you want a VPC
      based endpoint. Note that only SFTP and FTPS are supported on public endpoints.
  TransferSubnetIDs:
    Type: String
    Default: ''
    Description: Required if launching a VPC endpoint. Comma-seperated list of subnets that you would like
      the AWS Transfer endpoint to be provisioned into.
  TransferVPCID:
    Type: String
    Default: ''
    Description: Required if launching a VPC endpoint. The VPC ID that you would like the AWS Transfer endpoint
      to be provisioned into.
Conditions:
  CreateServer:
    Fn::Equals:
      - Ref: CreateServer
      - 'true'
  NotCreateServer:
    Fn::Not:
      - Condition: CreateServer
  SecretsManagerRegionProvided:
    Fn::Not:
      - Fn::Equals:
          - Ref: SecretsManagerRegion
          - ''
  TransferVPCEndpoint:
    Fn::Equals:
      - Ref: TransferEndpointType
      - 'VPC'
Outputs:
  ServerId:
    Value:
      Fn::GetAtt: TransferServer.ServerId
    Condition: CreateServer
  StackArn:
    Value:
      Ref: AWS::StackId
  TransferIdentityProviderUrl:
    Description: URL to pass to AWS Transfer CreateServer call as part of optional IdentityProviderDetails
    Value:
      Fn::Join:
      - ''
      - - https://
        - Ref: CustomIdentityProviderApi
        - .execute-api.
        - Ref: AWS::Region
        - .amazonaws.com/
        - Ref: ApiStage
    Condition: NotCreateServer
  TransferIdentityProviderInvocationRole:
    Description: IAM Role to pass to AWS Transfer CreateServer call as part of optional IdentityProviderDetails
    Value:
      Fn::GetAtt: TransferIdentityProviderRole.Arn
    Condition: NotCreateServer
Resources:
  TransferServer:
    Type: AWS::Transfer::Server
    Condition: CreateServer
    Properties:
      EndpointType:
        Ref: TransferEndpointType
      EndpointDetails:
        Fn::If:
          - TransferVPCEndpoint
          - SubnetIds:
              Fn::Split: [',', Ref: TransferSubnetIDs]
            VpcId:
              Ref: TransferVPCID
          - Ref: AWS::NoValue
      IdentityProviderDetails:
        InvocationRole:
          Fn::GetAtt: TransferIdentityProviderRole.Arn
        Url:
          Fn::Join:
            - ''
            - - https://
              - Ref: CustomIdentityProviderApi
              - .execute-api.
              - Ref: AWS::Region
              - .amazonaws.com/
              - Ref: ApiStage
      IdentityProviderType: API_GATEWAY
      LoggingRole:
        Fn::GetAtt: TransferCWLoggingRole.Arn
  TransferCWLoggingRole:
    Description: IAM role used by Transfer to log API requests to CloudWatch
    Type: AWS::IAM::Role
    Condition: CreateServer
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - transfer.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSTransferLoggingAccess
  CustomIdentityProviderApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: Transfer Family Secrets Manager Integration API
      Description: API used for Transfer Family to access user information in Secrets Manager
      FailOnWarnings: true
      EndpointConfiguration:
        Types:
        - REGIONAL
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      ManagedPolicyArns:
      - Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
      - PolicyName: LambdaSecretsPolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - secretsmanager:GetSecretValue
            Resource:
              Fn::Sub:
                - arn:${AWS::Partition}:secretsmanager:${SecretsRegion}:${AWS::AccountId}:secret:s-*
                - SecretsRegion:
                    Fn::If:
                      - SecretsManagerRegionProvided
                      - Ref: SecretsManagerRegion
                      - Ref: AWS::Region
  ApiCloudWatchLogsRole:
    Description: IAM role used by API Gateway to log API requests to CloudWatch
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - apigateway.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
      - PolicyName: ApiGatewayLogsPolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:DescribeLogGroups
            - logs:DescribeLogStreams
            - logs:PutLogEvents
            - logs:GetLogEvents
            - logs:FilterLogEvents
            Resource: "*"
  ApiLoggingAccount:
    Type: AWS::ApiGateway::Account
    DependsOn:
    - CustomIdentityProviderApi
    Properties:
      CloudWatchRoleArn:
        Fn::GetAtt: ApiCloudWatchLogsRole.Arn
  ApiStage:
    Type: AWS::ApiGateway::Stage
    Properties:
      DeploymentId:
        Ref: ApiDeployment202008
      MethodSettings:
      - DataTraceEnabled: false
        HttpMethod: "*"
        LoggingLevel: INFO
        ResourcePath: "/*"
      RestApiId:
        Ref: CustomIdentityProviderApi
      StageName: prod
  ApiDeployment202008:
    DependsOn:
    - GetUserConfigRequest
    Type: AWS::ApiGateway::Deployment
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
  TransferIdentityProviderRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: transfer.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
      - PolicyName: TransferCanInvokeThisApi
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - execute-api:Invoke
            Resource:
              Fn::Sub: arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${CustomIdentityProviderApi}/prod/GET/*
      - PolicyName: TransferCanReadThisApi
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - apigateway:GET
            Resource: "*"
  GetUserConfigLambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Description: A function to lookup and return user data from AWS Secrets Manager.
      Handler: index.lambda_handler
      Role:
        Fn::GetAtt: LambdaExecutionRole.Arn
      Runtime: python3.7
      Environment:
        Variables:
          SecretsManagerRegion:
            Fn::If:
              - SecretsManagerRegionProvided
              - Ref: SecretsManagerRegion
              - Ref: AWS::Region
  GetUserConfigLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:invokeFunction
      FunctionName:
        Fn::GetAtt: GetUserConfigLambda.Arn
      Principal: apigateway.amazonaws.com
      SourceArn:
        Fn::Sub: arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${CustomIdentityProviderApi}/*
  ServersResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Fn::GetAtt:
        - CustomIdentityProviderApi
        - RootResourceId
      PathPart: servers
  ServerIdResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Ref: ServersResource
      PathPart: "{serverId}"
  UsersResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Ref: ServerIdResource
      PathPart: users
  UserNameResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Ref: UsersResource
      PathPart: "{username}"
  GetUserConfigResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Ref: UserNameResource
      PathPart: config
  GetUserConfigRequest:
    Type: AWS::ApiGateway::Method
    DependsOn: GetUserConfigResponseModel
    Properties:
      AuthorizationType: AWS_IAM
      HttpMethod: GET
      Integration:
        Type: AWS
        IntegrationHttpMethod: POST
        Uri:
          Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - ":apigateway:"
              - Ref: AWS::Region
              - ":lambda:path/2015-03-31/functions/"
              - Fn::GetAtt:
                - GetUserConfigLambda
                - Arn
              - "/invocations"
        IntegrationResponses:
        - StatusCode: 200
        RequestTemplates:
          application/json: |
            {
              "username": "$util.urlDecode($input.params('username'))",
              "password": "$util.escapeJavaScript($input.params('Password')).replaceAll("\\'","'")",
              "protocol": "$input.params('protocol')",
              "serverId": "$input.params('serverId')",
              "sourceIp": "$input.params('sourceIp')"
            }
      RequestParameters:
        method.request.header.Password: false
      ResourceId:
        Ref: GetUserConfigResource
      RestApiId:
        Ref: CustomIdentityProviderApi
      MethodResponses:
      - StatusCode: 200
        ResponseModels:
          application/json: UserConfigResponseModel
  GetUserConfigResponseModel:
    Type: AWS::ApiGateway::Model
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ContentType: application/json
      Description: API response for GetUserConfig
      Name: UserConfigResponseModel
      Schema:
        "$schema": http://json-schema.org/draft-04/schema#
        title: UserUserConfig
        type: object
        properties:
          HomeDirectory:
            type: string
          Role:
            type: string
          Policy:
            type: string
          PublicKeys:
            type: array
            items:
              type: string
修正後 template.yml

template.yml

---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: A basic template for creating a Lambda-backed API Gateway for use as
  a custom identity provider in AWS Transfer. It authenticates against an
  entry in AWS Secrets Manager of the format server-id/username. Additionaly, the secret
  must hold the key-value pairs for all user properties returned to AWS Transfer.
  If you choose to also launch the AWS Transfer endpoint it will be provisioned with the SFTP protocol.
  This can be changed after launch. Note that FTP is only supported on VPC endpoints
  You can modify the Lambda function code to do something different after deployment.
Parameters:
  CreateServer:
    AllowedValues:
      - 'true'
      - 'false'
    Type: String
    Description: Whether this stack creates an AWS Transfer endpoint or not. If the endpoint is created as
      part of the stack, the custom identity provider is automatically associated with it.
    Default: 'true'
  SecretsManagerRegion:
    Type: String
    Description: (Optional) The region the secrets are stored in. If this value is not provided, the
      region this stack is deployed in will be used. Use this field if you are deploying this stack in
      a region where SecretsMangager is not available.
    Default: ''
  # TransferEndpointType:
  #   AllowedValues:
  #     - 'PUBLIC'
  #     - 'VPC'
  #   Type: String
  #   Default: 'PUBLIC'
  #   Description: Select PUBLIC if you want a public facing AWS Transfer endpoint or VPC if you want a VPC
  #     based endpoint. Note that only SFTP and FTPS are supported on public endpoints.
  TransferDomain:
    AllowedValues:
      - 'EFS'
      - 'S3'
    Type: String
    Default: 'EFS'
    Description: Specifies the domain of the storage system that is used for file transfers.
  TransferSubnetIDs:
    Type: String
    Default: ''
    Description: Required if launching a VPC endpoint. Comma-seperated list of subnets that you would like
      the AWS Transfer endpoint to be provisioned into.
  SecurityGroupIds:
    Type: String
    Default: ''
    Description: Required if launching a VPC endpoint. 
  ElasticIP:
    Type: String
    Default: ''
    Description: Required if launching a VPC endpoint. 
  TransferVPCID:
    Type: String
    Default: ''
    Description: Required if launching a VPC endpoint. The VPC ID that you would like the AWS Transfer endpoint
      to be provisioned into.
Conditions:
  CreateServer:
    Fn::Equals:
      - Ref: CreateServer
      - 'true'
  NotCreateServer:
    Fn::Not:
      - Condition: CreateServer
  SecretsManagerRegionProvided:
    Fn::Not:
      - Fn::Equals:
          - Ref: SecretsManagerRegion
          - ''
  # TransferVPCEndpoint:
  #   Fn::Equals:
  #     - Ref: TransferEndpointType
  #     - 'VPC'
Outputs:
  ServerId:
    Value:
      Fn::GetAtt: TransferServer.ServerId
    Condition: CreateServer
  StackArn:
    Value:
      Ref: AWS::StackId
  TransferIdentityProviderUrl:
    Description: URL to pass to AWS Transfer CreateServer call as part of optional IdentityProviderDetails
    Value:
      Fn::Join:
      - ''
      - - https://
        - Ref: CustomIdentityProviderApi
        - .execute-api.
        - Ref: AWS::Region
        - .amazonaws.com/
        - Ref: ApiStage
    Condition: NotCreateServer
  TransferIdentityProviderInvocationRole:
    Description: IAM Role to pass to AWS Transfer CreateServer call as part of optional IdentityProviderDetails
    Value:
      Fn::GetAtt: TransferIdentityProviderRole.Arn
    Condition: NotCreateServer
Resources:
  TransferServer:
    Type: AWS::Transfer::Server
    Condition: CreateServer
    Properties:
      Domain: EFS
      EndpointType: VPC
      # EndpointDetails:
      #   Fn::If:
      #     - TransferVPCEndpoint
      #     - SubnetIds:
      #         Fn::Split: [',', Ref: TransferSubnetIDs]
      #       VpcId:
      #         Ref: TransferVPCID
      #     - Ref: AWS::NoValue
      EndpointDetails:
        AddressAllocationIds:
          - Ref: ElasticIP
        SecurityGroupIds:
          - Ref: SecurityGroupIds
        SubnetIds:
          Fn::Split: [',', Ref: TransferSubnetIDs]
        VpcId: 
          Ref: TransferVPCID
      IdentityProviderDetails:
        InvocationRole:
          Fn::GetAtt: TransferIdentityProviderRole.Arn
        Url:
          Fn::Join:
            - ''
            - - https://
              - Ref: CustomIdentityProviderApi
              - .execute-api.
              - Ref: AWS::Region
              - .amazonaws.com/
              - Ref: ApiStage
      IdentityProviderType: API_GATEWAY
      LoggingRole:
        Fn::GetAtt: TransferCWLoggingRole.Arn
  TransferCWLoggingRole:
    Description: IAM role used by Transfer to log API requests to CloudWatch
    Type: AWS::IAM::Role
    Condition: CreateServer
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - transfer.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSTransferLoggingAccess
  CustomIdentityProviderApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: Transfer Family Secrets Manager Integration API
      Description: API used for Transfer Family to access user information in Secrets Manager
      FailOnWarnings: true
      EndpointConfiguration:
        Types:
        - REGIONAL
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      ManagedPolicyArns:
      - Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
      - PolicyName: LambdaSecretsPolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - secretsmanager:GetSecretValue
            Resource:
              Fn::Sub:
                - arn:${AWS::Partition}:secretsmanager:${SecretsRegion}:${AWS::AccountId}:secret:s-*
                - SecretsRegion:
                    Fn::If:
                      - SecretsManagerRegionProvided
                      - Ref: SecretsManagerRegion
                      - Ref: AWS::Region
  ApiCloudWatchLogsRole:
    Description: IAM role used by API Gateway to log API requests to CloudWatch
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - apigateway.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
      - PolicyName: ApiGatewayLogsPolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:DescribeLogGroups
            - logs:DescribeLogStreams
            - logs:PutLogEvents
            - logs:GetLogEvents
            - logs:FilterLogEvents
            Resource: "*"
  ApiLoggingAccount:
    Type: AWS::ApiGateway::Account
    DependsOn:
    - CustomIdentityProviderApi
    Properties:
      CloudWatchRoleArn:
        Fn::GetAtt: ApiCloudWatchLogsRole.Arn
  ApiStage:
    Type: AWS::ApiGateway::Stage
    Properties:
      DeploymentId:
        Ref: ApiDeployment202008
      MethodSettings:
      - DataTraceEnabled: false
        HttpMethod: "*"
        LoggingLevel: INFO
        ResourcePath: "/*"
      RestApiId:
        Ref: CustomIdentityProviderApi
      StageName: prod
  ApiDeployment202008:
    DependsOn:
    - GetUserConfigRequest
    Type: AWS::ApiGateway::Deployment
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
  TransferIdentityProviderRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: transfer.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
      - PolicyName: TransferCanInvokeThisApi
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - execute-api:Invoke
            Resource:
              Fn::Sub: arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${CustomIdentityProviderApi}/prod/GET/*
      - PolicyName: TransferCanReadThisApi
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - apigateway:GET
            Resource: "*"
  GetUserConfigLambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Description: A function to lookup and return user data from AWS Secrets Manager.
      Handler: index.lambda_handler
      Role:
        Fn::GetAtt: LambdaExecutionRole.Arn
      Runtime: python3.7
      Environment:
        Variables:
          SecretsManagerRegion:
            Fn::If:
              - SecretsManagerRegionProvided
              - Ref: SecretsManagerRegion
              - Ref: AWS::Region
  GetUserConfigLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:invokeFunction
      FunctionName:
        Fn::GetAtt: GetUserConfigLambda.Arn
      Principal: apigateway.amazonaws.com
      SourceArn:
        Fn::Sub: arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${CustomIdentityProviderApi}/*
  ServersResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Fn::GetAtt:
        - CustomIdentityProviderApi
        - RootResourceId
      PathPart: servers
  ServerIdResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Ref: ServersResource
      PathPart: "{serverId}"
  UsersResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Ref: ServerIdResource
      PathPart: users
  UserNameResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Ref: UsersResource
      PathPart: "{username}"
  GetUserConfigResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ParentId:
        Ref: UserNameResource
      PathPart: config
  GetUserConfigRequest:
    Type: AWS::ApiGateway::Method
    DependsOn: GetUserConfigResponseModel
    Properties:
      AuthorizationType: AWS_IAM
      HttpMethod: GET
      Integration:
        Type: AWS
        IntegrationHttpMethod: POST
        Uri:
          Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - ":apigateway:"
              - Ref: AWS::Region
              - ":lambda:path/2015-03-31/functions/"
              - Fn::GetAtt:
                - GetUserConfigLambda
                - Arn
              - "/invocations"
        IntegrationResponses:
        - StatusCode: 200
        RequestTemplates:
          application/json: |
            {
              "username": "$util.urlDecode($input.params('username'))",
              "password": "$util.escapeJavaScript($input.params('Password')).replaceAll("\\'","'")",
              "protocol": "$input.params('protocol')",
              "serverId": "$input.params('serverId')",
              "sourceIp": "$input.params('sourceIp')"
            }
      RequestParameters:
        method.request.header.Password: false
      ResourceId:
        Ref: GetUserConfigResource
      RestApiId:
        Ref: CustomIdentityProviderApi
      MethodResponses:
      - StatusCode: 200
        ResponseModels:
          application/json: UserConfigResponseModel
  GetUserConfigResponseModel:
    Type: AWS::ApiGateway::Model
    Properties:
      RestApiId:
        Ref: CustomIdentityProviderApi
      ContentType: application/json
      Description: API response for GetUserConfig
      Name: UserConfigResponseModel
      Schema:
        "$schema": http://json-schema.org/draft-04/schema#
        title: UserUserConfig
        type: object
        properties:
          HomeDirectory:
            type: string
          Role:
            type: string
          Policy:
            type: string
          PublicKeys:
            type: array
            items:
              type: string

1.2 Lambda（index.py）の修正

変更箇所　　

  • check_ipaddressメソッド
    • lookupの引数をAcceptedIpNetwork→AcceptedIpNetworkに変更（72~73行目）
  • build_responseメソッド
    • response_dataにPosixProfileを追加（151~154行目）
修正前 index.py

index.py

import os
import json
import boto3
import base64
from ipaddress import ip_network, ip_address
from botocore.exceptions import ClientError


def lambda_handler(event, context):
    # Get the required parameters
    required_param_list = ["serverId", "username", "protocol", "sourceIp"]
    for parameter in required_param_list:
        if parameter not in event:
            print("Incoming " + parameter + " missing - Unexpected")
            return {}

    input_serverId = event["serverId"]
    input_username = event["username"]
    input_protocol = event["protocol"]
    input_sourceIp = event["sourceIp"]
    input_password = event.get("password", "")

    print("ServerId: {}, Username: {}, Protocol: {}, SourceIp: {}"
          .format(input_serverId, input_username, input_protocol, input_sourceIp))

    # Check for password and set authentication type appropriately. No password means SSH auth
    print("Start User Authentication Flow")
    if input_password != "":
        print("Using PASSWORD authentication")
        authentication_type = "PASSWORD"
    else:
        if input_protocol == 'FTP' or input_protocol == 'FTPS':
            print("Empty password not allowed for FTP/S")
            return {}
        print("Using SSH authentication")
        authentication_type = "SSH"

    # Retrieve our user details from the secret. For all key-value pairs stored in SecretManager,
    # checking the protocol-specified secret first, then use generic ones.
    # e.g. If SFTPPassword and Password both exists, will be using SFTPPassword for authentication
    secret = get_secret(input_serverId + "/" + input_username)

    if secret is not None:
        secret_dict = json.loads(secret)
        # Run our password checks
        user_authenticated = authenticate_user(authentication_type, secret_dict, input_password, input_protocol)
        # Run sourceIp checks
        ip_match = check_ipaddress(secret_dict, input_sourceIp, input_protocol)

        if user_authenticated and ip_match:
            print("User authenticated, calling build_response with: " + authentication_type)
            return build_response(secret_dict, authentication_type, input_protocol)
        else:
            print("User failed authentication return empty response")
            return {}
    else:
        # Otherwise something went wrong. Most likely the object name is not there
        print("Secrets Manager exception thrown - Returning empty response")
        # Return an empty data response meaning the user was not authenticated
        return {}


def lookup(secret_dict, key, input_protocol):
    if input_protocol + key in secret_dict:
        print("Found protocol-specified {}".format(key))
        return secret_dict[input_protocol + key]
    else:
        return secret_dict.get(key, None)


def check_ipaddress(secret_dict, input_sourceIp, input_protocol):
    # accepted_ip_network = lookup(secret_dict, "AcceptedIpNetwork", input_protocol)
    accepted_ip_network = lookup(secret_dict, "AcceptedIPNetwork", input_protocol)
    if not accepted_ip_network:
        # No IP provided so skip checks
        print("No IP range provided - Skip IP check")
        return True

    net = ip_network(accepted_ip_network)
    if ip_address(input_sourceIp) in net:
        print("Source IP address match")
        return True
    else:
        print("Source IP address not in range")
        return False


def authenticate_user(auth_type, secret_dict, input_password, input_protocol):
    # Function returns True if: auth_type is password and passwords match or auth_type is SSH. Otherwise returns False
    if auth_type == "SSH":
        # Place for additional checks in future
        print("Skip password check as SSH login request")
        return True
    # auth_type could only be SSH or PASSWORD
    else:
        # Retrieve the password from the secret if exists
        password = lookup(secret_dict, "Password", input_protocol)
        if not password:
            print("Unable to authenticate user - No field match in Secret for password")
            return False

        if input_password == password:
            return True
        else:
            print("Unable to authenticate user - Incoming password does not match stored")
            return False


# Build out our response data for an authenticated response
def build_response(secret_dict, auth_type, input_protocol):
    response_data = {}
    # Check for each key value pair. These are required so set to empty string if missing
    role = lookup(secret_dict, "Role", input_protocol)
    if role:
        response_data["Role"] = role
    else:
        print("No field match for role - Set empty string in response")
        response_data["Role"] = ""

    # These are optional so ignore if not present
    policy = lookup(secret_dict, "Policy", input_protocol)
    if policy:
        response_data["Policy"] = policy

    # External Auth providers support chroot and virtual folder assignments so we'll check for that
    home_directory_details = lookup(secret_dict, "HomeDirectoryDetails", input_protocol)
    if home_directory_details:
        print("HomeDirectoryDetails found - Applying setting for virtual folders - "
              "Note: Cannot be used in conjunction with key: HomeDirectory")
        response_data["HomeDirectoryDetails"] = home_directory_details
        # If we have a virtual folder setup then we also need to set HomeDirectoryType to "Logical"
        print("Setting HomeDirectoryType to LOGICAL")
        response_data["HomeDirectoryType"] = "LOGICAL"

    # Note that HomeDirectory and HomeDirectoryDetails / Logical mode
    # can't be used together but we're not checking for this
    home_directory = lookup(secret_dict, "HomeDirectory", input_protocol)
    if home_directory:
        print("HomeDirectory found - Note: Cannot be used in conjunction with key: HomeDirectoryDetails")
        response_data["HomeDirectory"] = home_directory

    if auth_type == "SSH":
        public_key = lookup(secret_dict, "PublicKey", input_protocol)
        if public_key:
            response_data["PublicKeys"] = [public_key]
        else:
            # SSH Auth Flow - We don't have keys so we can't help
            print("Unable to authenticate user - No public keys found")
            return {}

    PosixProfile = lookup(secret_dict, "PosixProfile", input_protocol)
    if PosixProfile:
        print("Uid found")
        response_data["PosixProfile"] = [PosixProfile]

    return response_data


def get_secret(id):
    region = os.environ["SecretsManagerRegion"]
    print("Secrets Manager Region: " + region)
    print("Secret Name: " + id)

    # Create a Secrets Manager client
    client = boto3.session.Session().client(service_name="secretsmanager", region_name=region)

    try:
        resp = client.get_secret_value(SecretId=id)
        # Decrypts secret using the associated KMS CMK.
        # Depending on whether the secret is a string or binary, one of these fields will be populated.
        if "SecretString" in resp:
            print("Found Secret String")
            return resp["SecretString"]
        else:
            print("Found Binary Secret")
            return base64.b64decode(resp["SecretBinary"])
    except ClientError as err:
        print("Error Talking to SecretsManager: " + err.response["Error"]["Code"] + ", Message: " +
              err.response["Error"]["Message"])
        return None
修正後 index.py

index.py

import os
import json
import boto3
import base64
from ipaddress import ip_network, ip_address
from botocore.exceptions import ClientError


def lambda_handler(event, context):
    # Get the required parameters
    required_param_list = ["serverId", "username", "protocol", "sourceIp"]
    for parameter in required_param_list:
        if parameter not in event:
            print("Incoming " + parameter + " missing - Unexpected")
            return {}

    input_serverId = event["serverId"]
    input_username = event["username"]
    input_protocol = event["protocol"]
    input_sourceIp = event["sourceIp"]
    input_password = event.get("password", "")

    print("ServerId: {}, Username: {}, Protocol: {}, SourceIp: {}"
          .format(input_serverId, input_username, input_protocol, input_sourceIp))

    # Check for password and set authentication type appropriately. No password means SSH auth
    print("Start User Authentication Flow")
    if input_password != "":
        print("Using PASSWORD authentication")
        authentication_type = "PASSWORD"
    else:
        if input_protocol == 'FTP' or input_protocol == 'FTPS':
            print("Empty password not allowed for FTP/S")
            return {}
        print("Using SSH authentication")
        authentication_type = "SSH"

    # Retrieve our user details from the secret. For all key-value pairs stored in SecretManager,
    # checking the protocol-specified secret first, then use generic ones.
    # e.g. If SFTPPassword and Password both exists, will be using SFTPPassword for authentication
    secret = get_secret(input_serverId + "/" + input_username)

    if secret is not None:
        secret_dict = json.loads(secret)
        # Run our password checks
        user_authenticated = authenticate_user(authentication_type, secret_dict, input_password, input_protocol)
        # Run sourceIp checks
        ip_match = check_ipaddress(secret_dict, input_sourceIp, input_protocol)

        if user_authenticated and ip_match:
            print("User authenticated, calling build_response with: " + authentication_type)
            return build_response(secret_dict, authentication_type, input_protocol)
        else:
            print("User failed authentication return empty response")
            return {}
    else:
        # Otherwise something went wrong. Most likely the object name is not there
        print("Secrets Manager exception thrown - Returning empty response")
        # Return an empty data response meaning the user was not authenticated
        return {}


def lookup(secret_dict, key, input_protocol):
    if input_protocol + key in secret_dict:
        print("Found protocol-specified {}".format(key))
        return secret_dict[input_protocol + key]
    else:
        return secret_dict.get(key, None)


def check_ipaddress(secret_dict, input_sourceIp, input_protocol):
    # accepted_ip_network = lookup(secret_dict, "AcceptedIpNetwork", input_protocol)
    accepted_ip_network = lookup(secret_dict, "AcceptedIPNetwork", input_protocol)
    if not accepted_ip_network:
        # No IP provided so skip checks
        print("No IP range provided - Skip IP check")
        return True

    net = ip_network(accepted_ip_network)
    if ip_address(input_sourceIp) in net:
        print("Source IP address match")
        return True
    else:
        print("Source IP address not in range")
        return False


def authenticate_user(auth_type, secret_dict, input_password, input_protocol):
    # Function returns True if: auth_type is password and passwords match or auth_type is SSH. Otherwise returns False
    if auth_type == "SSH":
        # Place for additional checks in future
        print("Skip password check as SSH login request")
        return True
    # auth_type could only be SSH or PASSWORD
    else:
        # Retrieve the password from the secret if exists
        password = lookup(secret_dict, "Password", input_protocol)
        if not password:
            print("Unable to authenticate user - No field match in Secret for password")
            return False

        if input_password == password:
            return True
        else:
            print("Unable to authenticate user - Incoming password does not match stored")
            return False


# Build out our response data for an authenticated response
def build_response(secret_dict, auth_type, input_protocol):
    response_data = {}
    # Check for each key value pair. These are required so set to empty string if missing
    role = lookup(secret_dict, "Role", input_protocol)
    if role:
        response_data["Role"] = role
    else:
        print("No field match for role - Set empty string in response")
        response_data["Role"] = ""

    # These are optional so ignore if not present
    policy = lookup(secret_dict, "Policy", input_protocol)
    if policy:
        response_data["Policy"] = policy

    # External Auth providers support chroot and virtual folder assignments so we'll check for that
    home_directory_details = lookup(secret_dict, "HomeDirectoryDetails", input_protocol)
    if home_directory_details:
        print("HomeDirectoryDetails found - Applying setting for virtual folders - "
              "Note: Cannot be used in conjunction with key: HomeDirectory")
        response_data["HomeDirectoryDetails"] = home_directory_details
        # If we have a virtual folder setup then we also need to set HomeDirectoryType to "Logical"
        print("Setting HomeDirectoryType to LOGICAL")
        response_data["HomeDirectoryType"] = "LOGICAL"

    # Note that HomeDirectory and HomeDirectoryDetails / Logical mode
    # can't be used together but we're not checking for this
    home_directory = lookup(secret_dict, "HomeDirectory", input_protocol)
    if home_directory:
        print("HomeDirectory found - Note: Cannot be used in conjunction with key: HomeDirectoryDetails")
        response_data["HomeDirectory"] = home_directory

    if auth_type == "SSH":
        public_key = lookup(secret_dict, "PublicKey", input_protocol)
        if public_key:
            response_data["PublicKeys"] = [public_key]
        else:
            # SSH Auth Flow - We don't have keys so we can't help
            print("Unable to authenticate user - No public keys found")
            return {}

    PosixProfile = lookup(secret_dict, "PosixProfile", input_protocol)
    if PosixProfile:
        print("Uid found")
        response_data["PosixProfile"] = [PosixProfile]

    return response_data


def get_secret(id):
    region = os.environ["SecretsManagerRegion"]
    print("Secrets Manager Region: " + region)
    print("Secret Name: " + id)

    # Create a Secrets Manager client
    client = boto3.session.Session().client(service_name="secretsmanager", region_name=region)

    try:
        resp = client.get_secret_value(SecretId=id)
        # Decrypts secret using the associated KMS CMK.
        # Depending on whether the secret is a string or binary, one of these fields will be populated.
        if "SecretString" in resp:
            print("Found Secret String")
            return resp["SecretString"]
        else:
            print("Found Binary Secret")
            return base64.b64decode(resp["SecretBinary"])
    except ClientError as err:
        print("Error Talking to SecretsManager: " + err.response["Error"]["Code"] + ", Message: " +
              err.response["Error"]["Message"])
        return None

2. SAMの実行

いよいよ修正したテンプレートを実行します。

「SAM実行したことないから、よくわからない...」という方はこちらのブログが参考になります。（私もはじめてSAMを使いました。）

今回はAWS SAMをインストールするのがめんどうだったので、CloudShellから実行します。
※自端末にAWS SAMがインストールされている方は[2.1. SAMの実行環境作成]はとばして下さい。

2.1. SAMの実行環境作成

CloudShell起動後、sam initコマンドを実行し、SAMの実行環境を作成します。
以下の内容が対話方式で聞かれるため順に答えていきます。

Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location

1 - AWS Quick Start Templatesを選択。

Choose an AWS Quick Start application template
        1 - Hello World Example
        2 - Multi-step workflow
        3 - Serverless API
        4 - Scheduled task
        5 - Standalone function
        6 - Data processing
        7 - Infrastructure event management
        8 - Lambda EFS example
        9 - Machine Learning

1 - Hello World Exampleを選択。

Use the most popular runtime and package type? (Python and zip) [y/N]:

yを選択。

Would you like to enable X-Ray tracing on the function(s) in your application?  [y/N]:

Nを選択。

Project name [sam-app]:

そのままEnterを押下。ここまででSAMの実行環境が作成されます。

次に修正した以下のファイルをCloudShellにアップロードします。

  • template.yml
  • index.py

アップロードしたファイルを下記の場所に移動します。

sam-app
∟ template.yml
∟ src
　　　　　∟ index.py

2.2. SAMの実行

sam app配下でsam deploy -gコマンドを実行します。
デプロイ中にSAM CLI によるプロンプトを用いたガイドを有効になるため、ガイドに従ってオプションを入力します。

Stack Name []: test-transferFamily-CloudFormation(任意のスタック名)
AWS Region []: ap-northeast-1
Parameter CreateServer [true]: そのままEnterを押下
Parameter SecretsManagerRegion []: そのままEnterを押下
Parameter TransferDomain [EFS]: そのままEnterを押下
Parameter TransferSubnetIDs []: subnet-xxxxxxxx(作成先のSubnetID)
Parameter SecurityGroupIds []: sg-xxxxxxxx(設定するSecurityGroupId)
Parameter EIP []: eipalloc-xxxxxxxxxx(設定するEIPの割り当てID)
Parameter TransferVPCID []: vpc-xxxxxxxx(作成先のVPCID)
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy Confirm changes before deploy [Y/n]: Y
#SAM needs permission to be able to create roles to connect to the resources in your template Allow SAM CLI IAM role creation [Y/n]: Y
#Preserves the state of previously provisioned resources when an operation fails Disable rollback [Y/n]: Y
Save arguments to configuration file [Y/n]: Y
SAM configuration file [samconfig.toml]: そのままEnterを押下
SAM configuration environment [default]: そのままEnterを押下

デプロイするリソースの一覧が表示される

Deploy this changeset? [y/N]: y

ここでCloudFormationが実行されます。CloudShellにSuccessfully created/updated stack - in ap-northeast-1が表示されることを確認します。

3. SecretsManagerでシークレットを作成

最後に接続するためのユーザーを作成します。
SecretsManagerサービスから、[新しいシークレットを保存する]を選択します。

設定ステップ 1 : シークレットのタイプを選択
以下を選択し、[次]を押下します。

  • シークレットのタイプ: その他のシークレットタイプ
  • キー/値のペア: プレーンテキスト（値は下記参照）
    • "Password": "xxxxxxx" （任意のパスワード）
    • "PublicKey": "ssh-rsa xxxxxxx" （tarnsfer_key.pub（公開鍵）の中身）
    • "Role": "arn:aws:iam::xxxxxxxxxxx:role/test-transferfamily-role" （作成したIAMロールARN）
    • "HomeDirectory": "/fs-xxxxxxxxxxxxx" （"/" + efsのファイルID）
    • "SFTPAcceptedIPNetwork": "0.0.0.0/0" （今回はセキュリティグループでIPを制限しているため、0.0.0.0/0にしています。Transfer Familyのエンドポイントタイプをパブリックエンドポイントにした場合は、ここでIP範囲を制限することができます。）
    • "PosixProfile":
      • "Uid": "0" (0 に設定すると、ルートアクセスになります)
      • "Gid": "0" (0 に設定すると、ルートアクセスになります)

プレーンテキストの設定値例

{
  "Password": "xxxxxxx",
  "PublicKey": "ssh-rsa xxxxxxx",
  "Role": "arn:aws:iam::xxxxxxxxxxx:role/test-transferfamily-role",
  "HomeDirectory": "/fs-xxxxxxxxxxxxx",
  "SFTPAcceptedIPNetwork": "0.0.0.0",
  "PosixProfile": {
    "Uid": "0",
    "Gid": "0"
  }
}
  • 暗号化キー: aws/secretsmanager

設定ステップ 2 : シークレットを設定
以下を選択し、[次]を押下します。

  • シークレットの名前: s-xxxxxxxxx/test-transferfamily-user(TransferFamilyのサーバーID/接続するユーザ名)

設定ステップ 3: ローテーションを設定 - オプション
ここではシークレットのローテーションを設定できます。
今回は設定しないためそのまま[次]を押下します。

設定ステップ 4: レビュー
設定した値に誤りがないか確認後、[保存]を押下します。

シークレットが作成されたことを確認します。

接続してみた

最後に接続できるか確認してみます。

接続前に必要なTransferFamilyのエンドポイント名を控えておきます。

今回はSFTPクライアントソフトとしてCyberduckを使用して接続してみます。

最初は鍵認証で接続します。

  • プロトコル: SFTP
  • サーバ: s-xxxxxxx.amazonaws.com(TransferFamilyのエンドポイント)
  • ポート: 22
  • ユーザー名: test-transferfamily-user(SecretManagerで設定したユーザー名)
  • SSH Private Key: test-transferfamily-key（作成した秘密鍵）

はじめて接続する場合はこのような画面が表示されます。[許可]を選択します。

問題なく接続できました！

次にパスワード認証で接続します。

  • プロトコル: SFTP
  • サーバ: s-xxxxxxx.amazonaws.com(TransferFamilyのエンドポイント)
  • ポート: 22
  • ユーザー名: test-transferfamily-user(SecretManagerで設定したユーザー名)
  • パスワード: xxxxxxxxxxx（SecretManagerで設定したパスワード）

こちらも問題なく接続できました！

ファイルのアップロードもダウンロードも問題なくできました。

おまけ：特定IPのみ接続を許可するように変更する

今回構築に使用したLambdaはIP範囲での制限をかけることができるようになってます。
特定のIPのみ接続を許可するように設定を変更する場合はLambdaとSecretManagerを以下のように修正します。

TransferFamilyをパブリックエンドポイントで構築した場合は、セキュリティグループは使用できないため、こちらを使用してみましょう。

Lambda（index.py）の修正

変更箇所　　

  • check_ipaddressメソッド
    • 配列からIPリストを取得するように変更（71～85行目をコメントアウト。88行目にコードを追記）
修正後 index.py 
import os
import json
import boto3
import base64
from ipaddress import ip_network, ip_address
from botocore.exceptions import ClientError


def lambda_handler(event, context):
    # Get the required parameters
    required_param_list = ["serverId", "username", "protocol", "sourceIp"]
    for parameter in required_param_list:
        if parameter not in event:
            print("Incoming " + parameter + " missing - Unexpected")
            return {}

    input_serverId = event["serverId"]
    input_username = event["username"]
    input_protocol = event["protocol"]
    input_sourceIp = event["sourceIp"]
    input_password = event.get("password", "")

    print("ServerId: {}, Username: {}, Protocol: {}, SourceIp: {}"
          .format(input_serverId, input_username, input_protocol, input_sourceIp))

    # Check for password and set authentication type appropriately. No password means SSH auth
    print("Start User Authentication Flow")
    if input_password != "":
        print("Using PASSWORD authentication")
        authentication_type = "PASSWORD"
    else:
        if input_protocol == 'FTP' or input_protocol == 'FTPS':
            print("Empty password not allowed for FTP/S")
            return {}
        print("Using SSH authentication")
        authentication_type = "SSH"

    # Retrieve our user details from the secret. For all key-value pairs stored in SecretManager,
    # checking the protocol-specified secret first, then use generic ones.
    # e.g. If SFTPPassword and Password both exists, will be using SFTPPassword for authentication
    secret = get_secret(input_serverId + "/" + input_username)

    if secret is not None:
        secret_dict = json.loads(secret)
        # Run our password checks
        user_authenticated = authenticate_user(authentication_type, secret_dict, input_password, input_protocol)
        # Run sourceIp checks
        ip_match = check_ipaddress(secret_dict, input_sourceIp, input_protocol)

        if user_authenticated and ip_match:
            print("User authenticated, calling build_response with: " + authentication_type)
            return build_response(secret_dict, authentication_type, input_protocol)
        else:
            print("User failed authentication return empty response")
            return {}
    else:
        # Otherwise something went wrong. Most likely the object name is not there
        print("Secrets Manager exception thrown - Returning empty response")
        # Return an empty data response meaning the user was not authenticated
        return {}


def lookup(secret_dict, key, input_protocol):
    if input_protocol + key in secret_dict:
        print("Found protocol-specified {}".format(key))
        return secret_dict[input_protocol + key]
    else:
        return secret_dict.get(key, None)


  # def check_ipaddress(secret_dict, input_sourceIp, input_protocol):
  #     # accepted_ip_network = lookup(secret_dict, "AcceptedIpNetwork", input_protocol)
  #     accepted_ip_network = lookup(secret_dict, "AcceptedIPNetwork", input_protocol)
  #     if not accepted_ip_network:
  #         # No IP provided so skip checks
  #         print("No IP range provided - Skip IP check")
  #         return True

  #     net = ip_network(accepted_ip_network)
  #     if ip_address(input_sourceIp) in net:
  #         print("Source IP address match")
  #         return True
  #     else:
  #         print("Source IP address not in range")
  #         return False


def check_ipaddress(secret_dict, input_sourceIp, input_protocol):
    # accepted_ip_network = lookup(secret_dict, "AcceptedIpNetwork", input_protocol)
    accepted_ip_network = lookup(secret_dict, "AcceptedIPNetwork", input_protocol)
    if not accepted_ip_network:
        # No IP provided so skip checks
        print("No IP range provided - Skip IP check")
        return True

    ALLOW_LIST = accepted_ip_network
    print(ALLOW_LIST)

    # net = ip_network(accepted_ip_network)
    for wl in ALLOW_LIST:
        ip_nw = wl
        if input_sourceIp == ip_nw:
            print(input_sourceIp)
            print("Source IP address match")
            return True
        else:
            print(input_sourceIp)
            print("Source IP address not match")
            return False


def authenticate_user(auth_type, secret_dict, input_password, input_protocol):
    # Function returns True if: auth_type is password and passwords match or auth_type is SSH. Otherwise returns False
    if auth_type == "SSH":
        # Place for additional checks in future
        print("Skip password check as SSH login request")
        return True
    # auth_type could only be SSH or PASSWORD
    else:
        # Retrieve the password from the secret if exists
        password = lookup(secret_dict, "Password", input_protocol)
        if not password:
            print("Unable to authenticate user - No field match in Secret for password")
            return False

        if input_password == password:
            return True
        else:
            print("Unable to authenticate user - Incoming password does not match stored")
            return False


# Build out our response data for an authenticated response
def build_response(secret_dict, auth_type, input_protocol):
    response_data = {}
    # Check for each key value pair. These are required so set to empty string if missing
    role = lookup(secret_dict, "Role", input_protocol)
    if role:
        response_data["Role"] = role
    else:
        print("No field match for role - Set empty string in response")
        response_data["Role"] = ""

    # These are optional so ignore if not present
    policy = lookup(secret_dict, "Policy", input_protocol)
    if policy:
        response_data["Policy"] = policy

    # External Auth providers support chroot and virtual folder assignments so we'll check for that
    home_directory_details = lookup(secret_dict, "HomeDirectoryDetails", input_protocol)
    if home_directory_details:
        print("HomeDirectoryDetails found - Applying setting for virtual folders - "
              "Note: Cannot be used in conjunction with key: HomeDirectory")
        response_data["HomeDirectoryDetails"] = home_directory_details
        # If we have a virtual folder setup then we also need to set HomeDirectoryType to "Logical"
        print("Setting HomeDirectoryType to LOGICAL")
        response_data["HomeDirectoryType"] = "LOGICAL"

    # Note that HomeDirectory and HomeDirectoryDetails / Logical mode
    # can't be used together but we're not checking for this
    home_directory = lookup(secret_dict, "HomeDirectory", input_protocol)
    if home_directory:
        print("HomeDirectory found - Note: Cannot be used in conjunction with key: HomeDirectoryDetails")
        response_data["HomeDirectory"] = home_directory

    if auth_type == "SSH":
        public_key = lookup(secret_dict, "PublicKey", input_protocol)
        if public_key:
            response_data["PublicKeys"] = [public_key]
        else:
            # SSH Auth Flow - We don't have keys so we can't help
            print("Unable to authenticate user - No public keys found")
            return {}

    PosixProfile = lookup(secret_dict, "PosixProfile", input_protocol)
    if PosixProfile:
        print("Uid found")
        response_data["PosixProfile"] = [PosixProfile]

    return response_data


def get_secret(id):
    region = os.environ["SecretsManagerRegion"]
    print("Secrets Manager Region: " + region)
    print("Secret Name: " + id)

    # Create a Secrets Manager client
    client = boto3.session.Session().client(service_name="secretsmanager", region_name=region)

    try:
        resp = client.get_secret_value(SecretId=id)
        # Decrypts secret using the associated KMS CMK.
        # Depending on whether the secret is a string or binary, one of these fields will be populated.
        if "SecretString" in resp:
            print("Found Secret String")
            return resp["SecretString"]
        else:
            print("Found Binary Secret")
            return base64.b64decode(resp["SecretBinary"])
    except ClientError as err:
        print("Error Talking to SecretsManager: " + err.response["Error"]["Code"] + ", Message: " +
              err.response["Error"]["Message"])
        return None

SecretManagerのシークレットの値

変更箇所　　

  • "SFTPAcceptedIPNetwork"
    • 配列をIPアドレスで記載する（※/32といったCidrの記載は不要）

プレーンテキストの設定値

{
  "Password": "xxxxxxx",
  "PublicKey": "ssh-rsa xxxxxxx",
  "Role": "arn:aws:iam::xxxxxxxxxxx:role/test-transferfamily-role",
  "HomeDirectory": "/fs-xxxxxxxxxxxxx",
  "SFTPAcceptedIPNetwork": [
    "特定のIPアドレス",
    "特定のIPアドレス"
  ],
  "PosixProfile": {
    "Uid": "0",
    "Gid": "0"
  }
}

リソースの削除

検証後は作成したリソースを削除しましょう。
特にTransfer Familyはサーバ停止しても、料金が発生してしまいます。
こちらの順番で削除すると、削除時にエラーが発生せず、スムーズに削除できます。

  • test-transferFamily-CloudFormationの削除（CloudShellで sam deleteを実行します）
  • EFSの削除
  • EIPの解放
  • セキュリティグループの削除
  • VPCの削除
  • IAMロールの削除

最後に

今回は鍵・パスワード認証によるTransfer Family を使用してSFTP 経由でファイルを EFS に直接アップロードする方法をご紹介しました。

EC2にマウントしていないEFSに直接ファイルをアップロードする方法の記事があまりなく、苦戦しながら構築したため、今回記事にしました。

どなたかのお役に立てれば幸いです。 最後までお読みいただきありがとうございました！

以上、おつまみ（＠AWS11077）でした！

関連記事

Enable password authentication for AWS Transfer Family using AWS Secrets Manager (updated) | AWS Storage Blog

AcceptedIpNetwork

[アップデート] AWS Transfer Family の対象ストレージに Amazon EFS が追加サポートされました | DevelopersIO

【2022年10月版】AWS Transfer Family の SFTP を使ってみた | DevelopersIO

AWS Transfer Family のSFTPをパスワード方式で接続してみた(IP制限有り) | DevelopersIO

AWS

関連記事

[レポート] AI が作成した画像をプロッタで紙に出力（描画）する ＃IOT205 #AWSreInvent

板倉舞

2023.12.01

[速報] 拡張イメージスキャンが CI/CD パイプラインに組み込めるようになりました！#AWSreInvent

たかくに

2023.12.01

[AWS re:Invent 2023 / 随時更新] Dr. Werner Vogels Keynoteで発表された新サービスまとめ #AWSreInvent

菊池修治

2023.12.01

[ワークショップ]Amazon Qを使ったビジネスアプリケーション開発のハンズオン #AIM369-NEW #AWSreInvent

Takaaki Kakei

2023.12.01