Observing How Billed Memory in Amazon Bedrock AgentCore Runtime V2 Shrinks After Peak Usage Using USAGE_LOGS

Introduction

On September 18, 2026, a new platform version, V2, was added to Amazon Bedrock AgentCore Runtime. In the previous article, we checked the improved cold start performance in V2.

https://dev.classmethod.jp/articles/amazon-bedrock-agentcore-runtime-v2-tried/

The official AWS blog introduces the memory allocation optimization in V2. In this article, I compared observation points inside the agent process with billing-side telemetry to observe how that optimization actually shows up in memory consumption and billing.

https://aws.amazon.com/jp/blogs/machine-learning/the-new-agentcore-runtime-elastic-optimized-and-consistently-fast-starts/

Test Environment

In AgentCore Runtime V2 in ap-northeast-1, I prepared two runtimes: one for basic measurement and one for session measurement that delivers USAGE_LOGS. Both are the same V2, and both deploy the ECR image agentcore-cpuinfo:mem-probe-v2 (the cpuinfo in the image name is carried over from the previous article; this article measures memory). Note that MB in this article means MiB (1 MB = 1024 × 1024 bytes).

Measurement Design

Billing Telemetry (USAGE_LOGS)

I obtained the billing telemetry as USAGE_LOGS. When you associate the Runtime ARN with a delivery source, metrics such as agent.runtime.memory.gb_hours.used are delivered to the log group at 1-second granularity, tagged with session.id. The setup procedure is as follows.

REGION="ap-northeast-1"
ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
RUNTIME_ARN="arn:aws:bedrock-agentcore:${REGION}:${ACCOUNT_ID}:runtime/<RUNTIME_ID>"
LOG_GROUP="/aws/vendedlogs/bedrock-agentcore/mem-probe-usage"

# 1. Create the log group (destination)
aws logs create-log-group --log-group-name "${LOG_GROUP}" --region ${REGION}

# 2. Create the delivery destination
aws logs put-delivery-destination \
  --name "mem-probe-usage-dest" \
  --delivery-destination-configuration \
    "{\"destinationResourceArn\":\"arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:${LOG_GROUP}:*\"}" \
  --output-format "json" --region ${REGION}

# 3. Create the delivery source (associate USAGE_LOGS with the runtime ARN)
aws logs put-delivery-source \
  --name "mem-probe-usage-source" \
  --resource-arn "${RUNTIME_ARN}" \
  --log-type "USAGE_LOGS" \
  --region ${REGION}

# 4. Create the delivery
aws logs create-delivery \
  --delivery-source-name "mem-probe-usage-source" \
  --delivery-destination-arn \
    "arn:aws:logs:${REGION}:${ACCOUNT_ID}:delivery-destination:mem-probe-usage-dest" \
  --region ${REGION}

With this setup, I confirmed that events containing resource_arn, event_timestamp, attributes, and metrics are output. attributes contained items such as session.id, and metrics contained items such as agent.runtime.memory.gb_hours.used. Log events appeared about 9 minutes after the measurement finished, and it took about 20 minutes until all 3 sessions' worth were available.

agent.runtime.memory.gb_hours.used is the usage for one second expressed in GB-hours, so it can be converted into the average billed memory equivalent for that second with billed memory (MB) ≈ mem_gb_hours_per_s × 3600 × 1024. The billed memory conversion values from here on are calculated with this formula.

Auxiliary Observation Inside the Container

To check how well the billed memory in USAGE_LOGS corresponds to the values visible from inside the container, that is, whether VmRSS or memory.current can be used as a proxy indicator for the billed value, I observed process and cgroup memory.

In the test /invocations, the heap size to allocate is specified with alloc_mb. It returns /proc/self/status and /sys/fs/cgroup/memory.current at three points: before allocation, after allocation, and after release. When sleep_sec is specified, it records VmRSS and memory.current after release at fixed intervals.

"""
AgentCore Runtime V2 課金メモリ観測エージェント

/invocations に渡せるフィールド:
  alloc_mb    : ヒープ確保量 (MiB, default=0)
  sleep_sec   : 解放後にスリープする秒数 (default=0)
  interval_sec: sleep_sec > 0 の場合、interval_sec ごとに VmRSS と memory.current を記録 (default=10)

返却フィールド:
  vmrss          : /proc/self/status の VmRSS 増分（確保前後・解放後）
  cgroup_memory  : /sys/fs/cgroup/memory.current (存在しない場合は null)
  sleep_snapshots: sleep_sec > 0 の場合、interval_sec ごとの VmRSS と memory.current のスナップショット列
"""
import gc
import json
import os
import time
from flask import Flask, request, jsonify, Response

app = Flask(__name__)

CGROUP_MEMORY_CURRENT = "/sys/fs/cgroup/memory.current"
CGROUP_MEMORY_STAT    = "/sys/fs/cgroup/memory.stat"

def read_proc_status():
    """
    /proc/self/status からメモリ関連行を dict で返す。
    単位は kB のまま返す。
    """
    mem = {}
    try:
        with open("/proc/self/status") as f:
            for line in f:
                if ":" not in line:
                    continue
                key, _, val = line.partition(":")
                key = key.strip()
                if key.startswith("Vm") or key.startswith("Rss"):
                    mem[key] = val.strip()
    except Exception as e:
        mem["error"] = str(e)
    return mem

def read_cgroup_memory():
    """
    cgroup v2 の memory.current（コンテナ全体の現在メモリ使用量 bytes）を取得。
    存在しない場合は None。memory.stat の anon / file / shmem / kernel も補足する。
    """
    result = {}
    for path, key in [
        (CGROUP_MEMORY_CURRENT, "memory_current_bytes"),
    ]:
        try:
            with open(path) as f:
                raw = f.read().strip()
            result[key] = int(raw)
        except FileNotFoundError:
            result[key] = None
            result[f"{key}_note"] = f"not found: {path}"
        except Exception as e:
            result[key] = None
            result[f"{key}_error"] = str(e)
    # memory.stat から anon / file を補足
    try:
        stat = {}
        with open(CGROUP_MEMORY_STAT) as f:
            for line in f:
                parts = line.strip().split()
                if len(parts) == 2 and parts[0] in {"anon", "file", "shmem", "kernel"}:
                    stat[parts[0]] = int(parts[1])
        result["memory_stat"] = stat
    except FileNotFoundError:
        result["memory_stat"] = None
    except Exception as e:
        result["memory_stat_error"] = str(e)
    return result

def vmrss_kb():
    """VmRSS の現在値 (kB) だけを高速に取得。"""
    try:
        with open("/proc/self/status") as f:
            for line in f:
                if line.startswith("VmRSS:"):
                    return int(line.split()[1])
    except Exception:
        pass
    return 0

@app.route("/ping")
def ping():
    return Response("OK", status=200)

@app.route("/invocations", methods=["POST"])
def invocations():
    try:
        raw = request.data or b"{}"
        body = json.loads(raw)
    except Exception:
        body = {}

    alloc_mb     = int(body.get("alloc_mb", 0))
    sleep_sec    = int(body.get("sleep_sec", 0))
    interval_sec = int(body.get("interval_sec", 10))

    # --- 確保前 ---
    before        = read_proc_status()
    cgroup_before  = read_cgroup_memory()

    # --- ヒープ確保（ページイン保証） ---
    data = None
    alloc_error = None
    alloc_actual_bytes = 0
    t_alloc_start = time.monotonic()
    if alloc_mb > 0:
        try:
            data = bytearray(os.urandom(alloc_mb * 1024 * 1024))
            alloc_actual_bytes = len(data)
            _ = data[0]
            _ = data[-1]
        except Exception as e:
            alloc_error = str(e)
    t_alloc_ms = round((time.monotonic() - t_alloc_start) * 1000, 1)

    # --- 確保後 ---
    after         = read_proc_status()
    cgroup_after  = read_cgroup_memory()

    # --- 解放 ---
    del data
    gc.collect()

    # --- 解放後 ---
    freed         = read_proc_status()
    cgroup_freed  = read_cgroup_memory()

    # --- 解放後スリープ ---
    sleep_snapshots = []
    if sleep_sec > 0:
        t_sleep_start = time.monotonic()
        next_snap = interval_sec
        while True:
            elapsed = time.monotonic() - t_sleep_start
            if elapsed >= sleep_sec:
                break
            if elapsed >= next_snap:
                snap_rss = vmrss_kb()
                snap_cgroup = read_cgroup_memory()
                sleep_snapshots.append({
                    "elapsed_sec": round(elapsed, 1),
                    "vmrss_kb": snap_rss,
                    "cgroup_memory_current_bytes": snap_cgroup.get("memory_current_bytes"),
                })
                next_snap += interval_sec
            time.sleep(0.5)
        # 最終スナップ
        sleep_snapshots.append({
            "elapsed_sec": round(time.monotonic() - t_sleep_start, 1),
            "vmrss_kb": vmrss_kb(),
            "cgroup_memory_current_bytes": read_cgroup_memory().get("memory_current_bytes"),
        })

    def kb_val(d, key):
        v = d.get(key, "0 kB")
        try:
            return int(v.split()[0])
        except Exception:
            return 0

    rss_before_kb = kb_val(before, "VmRSS")
    rss_after_kb  = kb_val(after,  "VmRSS")
    rss_freed_kb  = kb_val(freed,  "VmRSS")

    return jsonify({
        "alloc_mb_requested":  alloc_mb,
        "alloc_actual_bytes":  alloc_actual_bytes,
        "alloc_time_ms":       t_alloc_ms,
        "alloc_error":         alloc_error,
        "vmrss": {
            "before_kb":       rss_before_kb,
            "after_kb":        rss_after_kb,
            "freed_kb":        rss_freed_kb,
            "delta_alloc_kb":  rss_after_kb - rss_before_kb,
            "delta_freed_kb":  rss_freed_kb - rss_after_kb,
        },
        "proc_status": {
            "before": before,
            "after":  after,
            "freed":  freed,
        },
        # --- cgroup ---
        "cgroup_memory": {
            "before": cgroup_before,
            "after":  cgroup_after,
            "freed":  cgroup_freed,
        },
        # --- スリープ中の推移 ---
        "sleep_snapshots": sleep_snapshots,
    })

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Observation Results from Inside the Container

Within the same session, I invoked with alloc_mb set to 0, 100, 500, and 1000.

The increase in VmRSS almost matched the allocated amount (the difference is in units of page size), and the amount of memory actually used by the agent process could be read from VmRSS. The process baseline was about 33MB.

The cgroup's /sys/fs/cgroup/memory.current also tracked the allocated amount.

However, the baseline of about 22MB was lower than the roughly 33MB of VmRSS. The breakdown of memory.stat read at the same time was anon=21.5MB, file=0, shmem=0, so in this measurement only anonymous pages were counted. In any case, the billed memory seen later is about 1,292MB even when idle, an order of magnitude different, and did not correspond to memory.current (about 22MB). Within the scope of this observation, memory.current could not be used as a proxy indicator for billed memory.

Behavior After Release and Stepwise Decline on the Billing Side

As shown in the freed column of the VmRSS table in the previous section, VmRSS returned to the pre-allocation level immediately after del and gc.collect(). The following table shows VmRSS and memory.current obtained every 10 seconds over the following 150 seconds (the sleep-test-a session after allocating and releasing alloc_mb=1000).

Throughout the 150 seconds, neither VmRSS nor memory.current changed in level. The reclamation behavior on the billing side was not visible from the observation points inside the container.

The billing side looked different. I ran the same sleep-test-a session (sleeping for 150 seconds after allocating alloc_mb=1000) concurrently with the other 2 sessions from 12:52 to 12:57 JST, and the following table shows its USAGE_LOGS converted to billed memory.

In this sleep-test-a session's measurement, the billed memory equivalent declined monotonically. Rather than dropping all at once at a particular milestone, it showed a stepwise decline, with intervals where the decline steepened around 77 seconds and around 192–219 seconds.

Cross-Checking with CloudWatch

The invoke response body did not include billing values. The billing-side values are obtained from CloudWatch and USAGE_LOGS. First, I checked MemoryUsed-GBHours at 1-minute granularity. The namespace was AWS/Bedrock-AgentCore, and the dimensions were Service=AgentCore.Runtime and Resource=<runtime ARN>. The values shown below were observed on the Runtime for basic measurement. This is a different time window and a different measurement from the 3 USAGE_LOGS sessions in the previous section, but I confirmed that the values for a V2 Runtime can also be retrieved from CloudWatch with the same namespace and dimensions.

These metrics appeared about 20 minutes after 12:18 JST, when I ended the measurement.

The USAGE_LOGS side had 1-second granularity. Events arrived in the following structure.

{
  "attributes": {
    "time_elapsed_seconds": 1.00,
    "session.id": "mem-probe-ext-seq-b22a9854-..."
  },
  "metrics": {
    "agent.runtime.memory.gb_hours.used": 0.000350483296511,
    "agent.runtime.vcpu.hours.used": 0.000556712777778
  }
}

Applying the conversion formula given in the measurement design to the event above, 0.000350 was about 1,292MB.

For the seq session, which ran concurrently with the other 2 sessions from 12:46 to 12:58 JST, converting the values during the operation period in which alloc_mb was changed in sequence into billed memory gives the following.

Immediately after session start, even at the point where nothing had been allocated, the billed memory was about 1,292MB. The process RSS (VmRSS) at that point was about 33MB, so the billed memory was about 39 times that.

When waiting in the idle state, the billed memory kept declining.

Even the roughly 796MB after 667 seconds was about 24 times the process RSS (about 33MB). The ratio ranged from 24 to 39 times depending on the session's elapsed time.

The totals for the 3 sessions (seq, which changes alloc_mb in sequence; sleep-ada21282, which is mostly sleeping; and sleep-test-a, which sleeps for 150 seconds) are as follows. "Observed duration" is the range for which USAGE_LOGS could be obtained, and total mem_gb_hours is the total for that period.

The heap was actually allocated and used for only a few tens of seconds or less in each session. Even so, billed memory continued to be counted in the idle state for as long as the session lasted, and in seq, 667 seconds' worth (equivalent to about 800–1,300MB) accumulated against about 15 seconds of actual work. Most of the billed memory came from the sustained idle portion rather than the allocation peak.

Summary

In V2, the memory billing rules changed and the unit price went up. On the other hand, when memory allocated at peak is released, the billable memory is also reclaimed during the session. AWS states that, depending on the amount of memory released, costs can be lower than with V1.

This time, I set up USAGE_LOGS on V2 and confirmed that billed memory keeps declining even after the peak when allocating 1000MB. By summing agent.runtime.memory.gb_hours.used per session or target period and multiplying by the unit price for each region and version, you can estimate memory charges before billing.

When using V2 at a certain scale or above, I recommend recording USAGE_LOGS during load tests that reproduce production operation from the evaluation stage, and carrying out a cost estimate for memory billing.