TV放送の情報を返すモックMCPサーバーをAgent Coreで構築してDevOps Agentのインシデント調査に活用できないか検証してみた

import { pipeline } from 'node:stream/promises';
import { Readable } from 'node:stream';

export const lambdaHandler = awslambda.streamifyResponse(async (event, responseStream, _context) => {
    const metadata = {
      statusCode: 200,
      headers: {
        "Content-Type": "text/event-stream; charset=utf-8",
      }
    };	
		responseStream = awslambda.HttpResponseStream.from(responseStream, metadata);
    const readable = Readable.from([
      "TV放送があったのは以下の日付です\n",
      "2026-01-01T10:00:00+09:00 ~ 2026-01-01T10:10:00+09:00 著名な芸能人によって紹介されました  \n",
      "2026-01-02T11:00:00+09:00 ~ 2026-01-02T11:10:00+09:00 著名なアイドルによって紹介されました  \n",
      "2026-01-07T13:20:00+09:00 ~ 2026-01-07T13:40:00+09:00  著名なタレントによって紹介されました  \n",
    ]);
    await pipeline(readable, responseStream);
});

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
  Domain:
    Type: String
    Description: Cognitoユーザープール用のドメイン
Globals:
  Function:
    Timeout: 600
    Tracing: Active
  Api:
    TracingEnabled: true
Resources:
  McpFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: mcp/
      Handler: app.lambdaHandler
      Runtime: nodejs24.x
      Architectures:
      - x86_64
      Events:
        mcp:
          Type: Api
          Properties:
            Path: /mcp
            Method: get
            RestApiId: !Ref McpApi
  McpApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName:
        prd
      DefinitionBody:
        openapi: "3.0.1"
        info:
          title: "stream-mcp"
          version: "1.0"
        servers: []
        paths:
          /mcp:
            get:
              summary: Get TV broadcast information
              description: |-
                Get information such as when it was broadcast on TV and who featured it.
              operationId: getTVBroadcastInfo
              x-amazon-apigateway-integration:
                type: "aws_proxy"
                uri:
                  Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2021-11-15/functions/${McpFunction.Arn}/response-streaming-invocations
                httpMethod: "POST"
                passthroughBehavior: "when_no_match"
                responseTransferMode: "STREAM"
                contentHandling: "CONVERT_TO_TEXT"
              responses:
                "200":
                  description: TV broadcast information retrieved successfully
                  content:
                    text/event-stream:
                      schema:
                        type: string
              security:
                - sigv4: []                        
        components:
          securitySchemes:
            sigv4:
              type: apiKey
              name: Authorization
              in: header
              x-amazon-apigateway-authtype: awsSigv4                        
  Gateway:
    Type: AWS::BedrockAgentCore::Gateway
    Properties:
      AuthorizerType: CUSTOM_JWT
      AuthorizerConfiguration:
        CustomJWTAuthorizer: 
          DiscoveryUrl: !Sub https://cognito-idp.${AWS::Region}.amazonaws.com/${AgentCoreGWUserPool}/.well-known/openid-configuration
          AllowedClients:
            - !Ref AgentCoreGWClient
      Description: DevOps Agentと連携させるためのゲートウェイ
      Name: gateway-for-devops-agent
      ProtocolType: MCP
      RoleArn: !GetAtt GatewayRole.Arn
      ExceptionLevel: DEBUG
  # GatewayTarget:
  #   Type: AWS::BedrockAgentCore::GatewayTarget
  #   Properties:
  #     CredentialProviderConfigurations: 
  #       - CredentialProviderType: GATEWAY_IAM_ROLE
  #     GatewayIdentifier: !Ref Gateway
  #     Name: api-gateway-target
  #     TargetConfiguration: 
  #     ターゲットにAPI GWを指定するのはCloud Formationがまだ未対応なので手動で設定する
  GatewayRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: bedrock-agentcore.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: GatewayRolePolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - execute-api:Invoke
                Resource: 
                  - !Sub arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${McpApi}/*
  AgentCoreGWUserPool:
    Type: AWS::Cognito::UserPool
  AgentCoreGWResourceServer:
    Type: AWS::Cognito::UserPoolResourceServer
    Properties:
      Identifier: gateway
      Name: gateway
      Scopes:
        - ScopeName: genesis-gateway:invoke
          ScopeDescription: Scope for invoking the genesis gateway
      UserPoolId: !Ref AgentCoreGWUserPool    
  AgentCoreGWClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      UserPoolId: !Ref AgentCoreGWUserPool
      GenerateSecret: true
      SupportedIdentityProviders: 
        - COGNITO
      AllowedOAuthFlows:
        - client_credentials
      AllowedOAuthFlowsUserPoolClient: true
      AllowedOAuthScopes:
        - gateway/genesis-gateway:invoke
    DependsOn: AgentCoreGWResourceServer
  AgentCoreGWUserPoolDomain:
    Type: AWS::Cognito::UserPoolDomain
    Properties:
      UserPoolId: !Ref AgentCoreGWUserPool
      Domain: !Ref Domain

curl -X POST https://<Cognitoのドメイン>.auth.ap-northeast-1.amazoncognito.com/oauth2/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials&client_id=<クライアントID>&client_secret=<クライアントシークレット>"

servers:
- url: "https://<API GWのID>.execute-api.ap-northeast-1.amazonaws.com/{basePath}"
  variables:
    basePath:
        default: "prd"

Default Run Book

Always use this Run Book

One of the characteristics of this system is that spikes in access can occur due to TV broadcasts.
TV broadcast information can be obtained through the MCP server.
When using the MCP server, the parameter `basePath` must be `prd`.

ab -n 50000 -c 200 https://<API GWのエンドポイント>

Excellent! I found highly relevant information.
The TV broadcast data shows a broadcast occurred on 2026-01-07 from 13:20-13:40 JST (04:20-04:40 UTC),
which aligns precisely with the incident time of 04:25 UTC. 
Now let me gather topology and metrics data:

TV broadcast triggered 50,000x traffic surge to PetStore API
A TV broadcast featuring a celebrity introduction aired on 2026-01-07 from 13:20-13:40 JST (04:20-04:40 UTC), 
directly causing a traffic surge to the PetStore API Gateway.
The API experienced 50,000 requests at 04:26:00 UTC, representing a 50,000x increase from the idle baseline (0 requests during 03:00-04:00 UTC period).
This TV broadcast traffic pattern is a documented characteristic of the system per the operational runbook. 
The API Gateway MOCK integration at GET / likely handled the traffic by returning static HTML welcome pages without backend processing,
evidenced by: 
  (1) zero Lambda invocations,
  (2) absence of integration latency metrics,
  and (3) exceptional performance with p99 latency under 5ms.
The system handled this surge successfully with zero errors (0 4XX, 0 5XX) and no throttling, demonstrating the API Gateway infrastructure scaled seamlessly to handle the cold-start traffic spike.

The PetStore API Gateway (<API GWのID>) experienced a massive traffic spike from 1 request/minute baseline to 50,000 requests at 2026-01-07T04:26:00Z.
This surge coincides with a TV broadcast (13:20-13:40 JST / 04:20-04:40 UTC) featuring a celebrity introduction.
No 4XX or 5XX errors were observed during the spike.

The impact lasted approximately 15 minutes (2026-01-07T04:25:00Z - 2026-01-07T04:40:00Z).

1. TV broadcast triggered 50,000x traffic surge to PetStore API
A TV broadcast featuring a celebrity introduction aired on 2026-01-07 from 13:20-13:40 JST (04:20-04:40 UTC), directly causing a traffic surge to the PetStore API Gateway.
The API experienced 50,000 requests at 04:26:00 UTC, representing a 50,000x increase from the idle baseline (0 requests during 03:00-04:00 UTC period).

This TV broadcast traffic pattern is a documented characteristic of the system per the operational runbook. 
The API Gateway MOCK integration at GET / likely handled the traffic by returning static HTML welcome pages without backend processing,
evidenced by:
  (1) zero Lambda invocations,
  (2) absence of integration latency metrics,
  and (3) exceptional performance with p99 latency under 5ms.
The system handled this surge successfully with zero errors (0 4XX, 0 5XX) and no throttling,
demonstrating the API Gateway infrastructure scaled seamlessly to handle the cold-start traffic spike.

API Gateway access logging disabled prevents detailed request analysis
The hypothesis that a specific endpoint pattern or client behavior caused issues during the traffic surge could not be validated because API Gateway access logging is not configured for the PetStore API (<API GWのID>). 
Access logs would have revealed: 
  (1) which specific endpoints (/GET, /pets, /pets/{petId}) received the 50,000 requests,
  (2) client IP addresses and user agents to identify bot vs legitimate user traffic,
  (3) per-request latency to identify outliers beyond the p99 metric,
  (4) any authorization failures or malformed requests that might indicate attack patterns.

The log group /aws/apigateway/welcome exists but has received no data since September 2024, and stage configuration confirms accessLogSettings is not configured.
This gap prevented definitive determination of which integration (MOCK at /, HTTP proxy at /pets, or HTTP proxy at /pets/{petId}) handled the surge, though metrics strongly suggest the MOCK integration.