I observed in USAGE_LOGS the behavior of Amazon Bedrock AgentCore Runtime V2 billing memory shrinking after peak

Introduction

On September 18, 2026, a new platform version V2 was added to Amazon Bedrock AgentCore Runtime. In the previous article, we confirmed the improved cold start performance in V2.

https://dev.classmethod.jp/articles/amazon-bedrock-agentcore-runtime-v2-tried/

The official AWS blog introduces memory allocation optimizations in V2. In this article, we observed how those optimizations manifest in actual memory consumption and billing, by cross-referencing observation points inside the agent process with billing-side telemetry.

https://aws.amazon.com/jp/blogs/machine-learning/the-new-agentcore-runtime-elastic-optimized-and-consistently-fast-starts/

Verification Environment

We prepared two setups for AgentCore Runtime V2 in ap-northeast-1: one for basic measurement and one for session measurement delivering USAGE_LOGS. Both use the same V2, deploying ECR image agentcore-cpuinfo:mem-probe-v2 (the cpuinfo in the image name is carried over from the previous article; in this article we are measuring memory). Note that MB in this article refers to MiB (1 MB = 1024 × 1024 bytes).

Measurement Design

Billing Telemetry (USAGE_LOGS)

Billing telemetry was obtained as USAGE_LOGS. By linking a Runtime ARN to a delivery source, metrics such as agent.runtime.memory.gb_hours.used are delivered to a log group at 1-second granularity with session.id attached. The setup steps are as follows.

REGION="ap-northeast-1"
ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
RUNTIME_ARN="arn:aws:bedrock-agentcore:${REGION}:${ACCOUNT_ID}:runtime/<RUNTIME_ID>"
LOG_GROUP="/aws/vendedlogs/bedrock-agentcore/mem-probe-usage"

# 1. Create log group (destination)
aws logs create-log-group --log-group-name "${LOG_GROUP}" --region ${REGION}

# 2. Create delivery destination
aws logs put-delivery-destination \
  --name "mem-probe-usage-dest" \
  --delivery-destination-configuration \
    "{\"destinationResourceArn\":\"arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:${LOG_GROUP}:*\"}" \
  --output-format "json" --region ${REGION}

# 3. Create delivery source (link USAGE_LOGS to runtime ARN)
aws logs put-delivery-source \
  --name "mem-probe-usage-source" \
  --resource-arn "${RUNTIME_ARN}" \
  --log-type "USAGE_LOGS" \
  --region ${REGION}

# 4. Create delivery
aws logs create-delivery \
  --delivery-source-name "mem-probe-usage-source" \
  --delivery-destination-arn \
    "arn:aws:logs:${REGION}:${ACCOUNT_ID}:delivery-destination:mem-probe-usage-dest" \
  --region ${REGION}

With this configuration, we confirmed that events containing resource_arn, event_timestamp, attributes, and metrics are output. attributes includes items such as session.id, and metrics includes items such as agent.runtime.memory.gb_hours.used. Log events appeared approximately 9 minutes after measurement completion, and it took about 20 minutes for all 3 sessions to be available.

Since agent.runtime.memory.gb_hours.used represents usage over 1 second in GB-hours units, the billable memory for that 1-second average can be converted using billed memory (MB) ≈ mem_gb_hours_per_s × 3600 × 1024. All billed memory conversion values going forward are calculated using this formula.

Supplementary Observation Inside the Container

To verify how well the billed memory from USAGE_LOGS corresponds to values visible from inside the container—that is, whether VmRSS or memory.current can be used as proxy metrics for billing values—we observed the memory of the process and cgroup.

In the /invocations endpoint for verification, the amount of heap to allocate is specified with alloc_mb. It returns /proc/self/status and /sys/fs/cgroup/memory.current at three points: before allocation, after allocation, and after release. If sleep_sec is specified, VmRSS and memory.current are recorded at fixed intervals after release.

"""
AgentCore Runtime V2 Billed Memory Observation Agent

Fields that can be passed to /invocations:
  alloc_mb    : Heap allocation amount (MiB, default=0)
  sleep_sec   : Number of seconds to sleep after release (default=0)
  interval_sec: If sleep_sec > 0, record VmRSS and memory.current every interval_sec seconds (default=10)

Return fields:
  vmrss          : VmRSS delta from /proc/self/status (before/after allocation, after release)
  cgroup_memory  : /sys/fs/cgroup/memory.current (null if not present)
  sleep_snapshots: If sleep_sec > 0, sequence of VmRSS and memory.current snapshots every interval_sec seconds
"""
import gc
import json
import os
import time
from flask import Flask, request, jsonify, Response

app = Flask(__name__)

CGROUP_MEMORY_CURRENT = "/sys/fs/cgroup/memory.current"
CGROUP_MEMORY_STAT    = "/sys/fs/cgroup/memory.stat"

def read_proc_status():
    """
    Return memory-related lines from /proc/self/status as a dict.
    Units are returned as-is in kB.
    """
    mem = {}
    try:
        with open("/proc/self/status") as f:
            for line in f:
                if ":" not in line:
                    continue
                key, _, val = line.partition(":")
                key = key.strip()
                if key.startswith("Vm") or key.startswith("Rss"):
                    mem[key] = val.strip()
    except Exception as e:
        mem["error"] = str(e)
    return mem

def read_cgroup_memory():
    """
    Get memory.current from cgroup v2 (current memory usage in bytes for the entire container).
    Returns None if not found. Also supplements anon / file / shmem / kernel from memory.stat.
    """
    result = {}
    for path, key in [
        (CGROUP_MEMORY_CURRENT, "memory_current_bytes"),
    ]:
        try:
            with open(path) as f:
                raw = f.read().strip()
            result[key] = int(raw)
        except FileNotFoundError:
            result[key] = None
            result[f"{key}_note"] = f"not found: {path}"
        except Exception as e:
            result[key] = None
            result[f"{key}_error"] = str(e)
    # Supplement anon / file from memory.stat
    try:
        stat = {}
        with open(CGROUP_MEMORY_STAT) as f:
            for line in f:
                parts = line.strip().split()
                if len(parts) == 2 and parts[0] in {"anon", "file", "shmem", "kernel"}:
                    stat[parts[0]] = int(parts[1])
        result["memory_stat"] = stat
    except FileNotFoundError:
        result["memory_stat"] = None
    except Exception as e:
        result["memory_stat_error"] = str(e)
    return result

def vmrss_kb():
    """Quickly retrieve only the current value of VmRSS (kB)."""
    try:
        with open("/proc/self/status") as f:
            for line in f:
                if line.startswith("VmRSS:"):
                    return int(line.split()[1])
    except Exception:
        pass
    return 0

@app.route("/ping")
def ping():
    return Response("OK", status=200)

@app.route("/invocations", methods=["POST"])
def invocations():
    try:
        raw = request.data or b"{}"
        body = json.loads(raw)
    except Exception:
        body = {}

    alloc_mb     = int(body.get("alloc_mb", 0))
    sleep_sec    = int(body.get("sleep_sec", 0))
    interval_sec = int(body.get("interval_sec", 10))

    # --- Before allocation ---
    before        = read_proc_status()
    cgroup_before  = read_cgroup_memory()

    # --- Heap allocation (page-in guaranteed) ---
    data = None
    alloc_error = None
    alloc_actual_bytes = 0
    t_alloc_start = time.monotonic()
    if alloc_mb > 0:
        try:
            data = bytearray(os.urandom(alloc_mb * 1024 * 1024))
            alloc_actual_bytes = len(data)
            _ = data[0]
            _ = data[-1]
        except Exception as e:
            alloc_error = str(e)
    t_alloc_ms = round((time.monotonic() - t_alloc_start) * 1000, 1)

    # --- After allocation ---
    after         = read_proc_status()
    cgroup_after  = read_cgroup_memory()

    # --- Release ---
    del data
    gc.collect()

    # --- After release ---
    freed         = read_proc_status()
    cgroup_freed  = read_cgroup_memory()

    # --- Post-release sleep ---
    sleep_snapshots = []
    if sleep_sec > 0:
        t_sleep_start = time.monotonic()
        next_snap = interval_sec
        while True:
            elapsed = time.monotonic() - t_sleep_start
            if elapsed >= sleep_sec:
                break
            if elapsed >= next_snap:
                snap_rss = vmrss_kb()
                snap_cgroup = read_cgroup_memory()
                sleep_snapshots.append({
                    "elapsed_sec": round(elapsed, 1),
                    "vmrss_kb": snap_rss,
                    "cgroup_memory_current_bytes": snap_cgroup.get("memory_current_bytes"),
                })
                next_snap += interval_sec
            time.sleep(0.5)
        # Final snapshot
        sleep_snapshots.append({
            "elapsed_sec": round(time.monotonic() - t_sleep_start, 1),
            "vmrss_kb": vmrss_kb(),
            "cgroup_memory_current_bytes": read_cgroup_memory().get("memory_current_bytes"),
        })

    def kb_val(d, key):
        v = d.get(key, "0 kB")
        try:
            return int(v.split()[0])
        except Exception:
            return 0

    rss_before_kb = kb_val(before, "VmRSS")
    rss_after_kb  = kb_val(after,  "VmRSS")
    rss_freed_kb  = kb_val(freed,  "VmRSS")

    return jsonify({
        "alloc_mb_requested":  alloc_mb,
        "alloc_actual_bytes":  alloc_actual_bytes,
        "alloc_time_ms":       t_alloc_ms,
        "alloc_error":         alloc_error,
        "vmrss": {
            "before_kb":       rss_before_kb,
            "after_kb":        rss_after_kb,
            "freed_kb":        rss_freed_kb,
            "delta_alloc_kb":  rss_after_kb - rss_before_kb,
            "delta_freed_kb":  rss_freed_kb - rss_after_kb,
        },
        "proc_status": {
            "before": before,
            "after":  after,
            "freed":  freed,
        },
        # --- cgroup ---
        "cgroup_memory": {
            "before": cgroup_before,
            "after":  cgroup_after,
            "freed":  cgroup_freed,
        },
        # --- Transition during sleep ---
        "sleep_snapshots": sleep_snapshots,
    })

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Observation Results from Inside the Container

We invoked within the same session while changing alloc_mb to 0, 100, 500, and 1000.

The VmRSS increase closely matched the allocated amount (differences are within page size units), and the actual memory used by the agent process could be read from VmRSS. The process baseline was approximately 33MB.

The cgroup's /sys/fs/cgroup/memory.current also tracked the allocated amount.

However, the baseline of approximately 22MB was below VmRSS's approximately 33MB. The breakdown of memory.stat read at the same time showed anon=21.5MB, file=0, shmem=0, meaning only anonymous pages were counted in this measurement. Regardless, the billed memory seen later was approximately 1,292MB even during idle—a completely different order of magnitude—and did not correspond to memory.current (approximately 22MB). Within the scope of this observation, memory.current could not be used as a proxy metric for billed memory.

Post-Release Behavior and Stepwise Decrease on the Billing Side

As shown in the freed column of the VmRSS table in the previous section, VmRSS immediately returned to the pre-allocation level right after del and gc.collect(). The following table shows VmRSS and memory.current values captured every 10 seconds over 150 seconds afterward (sleep-test-a session after allocating and releasing alloc_mb=1000).

Throughout the 150 seconds, neither VmRSS nor memory.current changed their levels. The billing-side reclamation behavior was not visible from the observation points inside the container.

The billing side showed a different picture. The same sleep-test-a session (150-second sleep after allocating alloc_mb=1000) was run simultaneously with 2 other sessions from 12:52 to 12:57 JST, and the following table shows those USAGE_LOGS converted to billed memory.

In this sleep-test-a session measurement, the billed memory equivalent value decreased monotonically and did not drop off at specific milestones all at once; instead, it showed a stepwise decrease with intervals of more rapid decline appearing around the 77-second mark and around the 192–219 second mark.

Cross-Referencing with CloudWatch

The billed values were not included in the invoke response body. Billing-side values are obtained from CloudWatch and USAGE_LOGS. First, we checked MemoryUsed-GBHours at 1-minute granularity. The namespace was AWS/Bedrock-AgentCore, with dimensions Service=AgentCore.Runtime and Resource=<runtime ARN>. The values shown below were observed from the Runtime used for basic measurement. While these are from a different time period and different measurement than the 3 sessions in USAGE_LOGS in the previous section, we confirmed that V2 Runtime values can also be obtained from CloudWatch using the same namespace and dimensions.

These metrics appeared approximately 20 minutes after measurement ended at 12:18 JST.

The USAGE_LOGS side was at 1-second granularity. Events arrived in the following structure.

{
  "attributes": {
    "time_elapsed_seconds": 1.00,
    "session.id": "mem-probe-ext-seq-b22a9854-..."
  },
  "metrics": {
    "agent.runtime.memory.gb_hours.used": 0.000350483296511,
    "agent.runtime.vcpu.hours.used": 0.000556712777778
  }
}

Applying the conversion formula shown in the measurement design section to the above event, 0.000350 was approximately 1,292MB.

For the seq session run simultaneously with 2 other sessions from 12:46 to 12:58 JST, converting the values during the operation period where alloc_mb was changed sequentially to billed memory yields the following.

Even immediately after session start with nothing allocated, the billed memory was approximately 1,292MB. At that point, the process RSS (VmRSS) was approximately 33MB, meaning billed memory was approximately 39 times that amount.

Remaining in idle state, the billed memory continued to decrease.

Even at approximately 796MB after 667 seconds, this was approximately 24 times the process RSS (approximately 33MB). The multiplier ranged between 24 and 39 times depending on session elapsed time.

The totals for the 3 sessions (seq changing alloc_mb sequentially, sleep-ada21282 focused on sleeping, and sleep-test-a with a 150-second sleep) are as follows. "Observed duration" refers to the range over which USAGE_LOGS could be obtained, and total mem_gb_hours is the total for that period.

The actual heap allocation and usage was within a few tens of seconds for each session. Nevertheless, billed memory continued to accumulate in an idle state for as long as the session lasted, and for seq, approximately 667 seconds' worth (equivalent to approximately 800–1,300MB) accumulated relative to approximately 15 seconds of actual work. The majority of billed memory came from the idle continuation period rather than the allocation peak.

Summary

In V2, the memory billing rules changed and the unit price increased. On the other hand, when memory allocated at peak is released, the billable memory is also reclaimed during the session. AWS advises that depending on the amount of memory released, costs can be kept lower than V1.

This time, we configured USAGE_LOGS in V2 and confirmed the behavior of billed memory decreasing even after the peak when allocating 1000MB. By summing agent.runtime.memory.gb_hours.used over a session or target period and multiplying by the unit price for each region and version, memory billing costs can be estimated before charges are incurred.

When using V2 at a certain scale or above, we recommend running load tests that replicate production workloads from the evaluation stage, recording USAGE_LOGS, and performing cost estimates for memory billing.