AWS Labsの「Bulk Executor for Amazon DynamoDB」でDynamoDBの一括操作をAWS Glueに任せてみる

Python 3.11.10
boto3 1.43.40
Bulk Executor v1 (main HEAD, 2026-07時点)
リージョン: ap-northeast-1

$ git clone https://github.com/awslabs/amazon-dynamodb-tools.git
$ cd amazon-dynamodb-tools/tools/bulk_executor
$ make install

$ ./bulk
Welcome to the Bulk Executor for Amazon DynamoDB!

This tool is useful for quickly executing bulk commands even large tables.
It uses AWS Glue internally to perform parallel execution.

Full documentation can be found at: https://github.com/awslabs/amazon-dynamodb-tools/blob/main/bulk_executor/README.md

Available actions found in client/src/python_modules:
  - bootstrap
  - copy
  - count
  - delete
  - diff
  - fill
  - find
  - load
  - load-export
  - revert-export
  - scancount
  - sql
  - teardown
  - update

$ ./bulk bootstrap --XRole READ-WRITE
Running action 'bootstrap' with arguments: {'verb': 'bootstrap', 'XRole': 'READ-WRITE'}
Adding Glue Job role...
READ-WRITE role pre-configured.
Bulk Executor Glue Job Role created: AWSGlueServiceRoleBulkDynamoDB-DdbReadWrite-ap-northeast-1
Creating CloudWatch log groups for Glue job...
Log group '/aws-glue/jobs/error' already exists.
Updated retention policy for existing log group /aws-glue/jobs/error
Log group '/aws-glue/jobs/output' already exists.
Updated retention policy for existing log group /aws-glue/jobs/output
Created Glue connection 'bulk-dynamodb-connection' for DynamoDB DataFrame source.
Attaching Glue Job Role arn:aws:iam::123456789012:role/AWSGlueServiceRoleBulkDynamoDB-DdbReadWrite-ap-northeast-1 to Bulk Executor Glue Job...
Bulk Executor Glue Job created successfully.
Bucket 'aws-glue-bulk-dynamodb-ap-northeast-1-123456789012-bn2aieu6r' created successfully!
Glue script 'server/src/root.py' uploaded into S3 successfully.
Successfully zipped .../server/src/python_modules to .../client/src/infrastructure/tmp/python_modules.zip using Python zipfile
Python modules archive 'client/src/infrastructure/tmp/python_modules.zip' uploaded into S3 successfully to 'server/src/python_modules.zip'.
Properties files 'server/src/log4j2.properties' uploaded into S3 successfully!

    Bulk Executor environment configurations:

      AWS -
        Account: 123456789012
        Region: ap-northeast-1
        Role: arn:aws:sts::123456789012:assumed-role/xxxxxxx/botocore-session-xxxxxxxxxx

$ aws glue get-job --job-name bulk_dynamodb --region ap-northeast-1 \
    --query 'Job.[Name,Role,GlueVersion,MaxRetries,Timeout,NumberOfWorkers,WorkerType,ExecutionClass]' \
    --output table
----------------------------------------------------------------------------------------------
|                                           GetJob                                           |
+--------------------------------------------------------------------------------------------+
|  bulk_dynamodb                                                                             |
|  arn:aws:iam::123456789012:role/AWSGlueServiceRoleBulkDynamoDB-DdbReadWrite-ap-northeast-1 |
|  5.1                                                                                       |
|  0                                                                                         |
|  60                                                                                        |
|  220                                                                                       |
|  G.1X                                                                                      |
|  None                                                                                      |
+--------------------------------------------------------------------------------------------+

$ aws s3 ls | grep aws-glue-bulk-dynamodb
2026-07-17 16:56:51 aws-glue-bulk-dynamodb-ap-northeast-1-123456789012-bn2aieu6r

$ aws iam list-roles \
    --query 'Roles[?starts_with(RoleName,`AWSGlueServiceRoleBulkDynamoDB`)].RoleName' \
    --output text
AWSGlueServiceRoleBulkDynamoDB-DdbReadWrite-ap-northeast-1

$ ./bulk scancount --help
usage: bulk scancount [-h] [--XExecutionClass {STANDARD,FLEX}]
                      [--XTimeout XTIMEOUT]
                      [--XNumberOfWorkers XNUMBEROFWORKERS]
                      [--XWorkerType {G.1X,G.2X,G.4X,G.8X,G.12X,G.16X,R.1X,R.2X,R.4X,R.8X}]
                      [--XWaitForDPU] [--XContinuousLogging]
                      [--XMaxWriteRate XMAXWRITERATE]
                      [--XMaxReadRate XMAXREADRATE] [--XAccount XACCOUNT]
                      [--XRegion XREGION] --table TABLE
                      [--filter-expression FILTER_EXPRESSION]
                      [--expression-names EXPRESSION_NAMES]
                      [--expression-values EXPRESSION_VALUES] [--index INDEX]

    Purpose of "scancount":
        Count items in a DynamoDB table, using a scan instead of a DynamoDB Connector (so usually faster)
        Required --table parameter
        Optional --index parameter if scanning a secondary index
        Optional --filter-expression parameter to specify a push-down FilterExpression predicate
        Optional --expression-names parameter to specify the expression names used in the filter-expression
        Optional --expression-values parameter to specify the expression values used in the filter-expression

    Examples:
        # Count all items in a table
        bulk scancount --table orders

        # Count using a filter expression (uses DynamoDB FilterExpression syntax)
        bulk scancount --table audit --filter-expression "#touched > :touched" --expression-names '{"#touched": "touched"}' --expression-values '{":touched":1742359403.0}'

$ aws dynamodb create-table \
    --table-name bulk-demo-202607 \
    --attribute-definitions AttributeName=pk,AttributeType=S AttributeName=sk,AttributeType=S \
    --key-schema AttributeName=pk,KeyType=HASH AttributeName=sk,KeyType=RANGE \
    --billing-mode PAY_PER_REQUEST \
    --region ap-northeast-1

$ aws dynamodb wait table-exists --table-name bulk-demo-202607 --region ap-northeast-1

$ aws dynamodb update-continuous-backups \
    --table-name bulk-demo-202607 \
    --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true \
    --region ap-northeast-1
{
    "ContinuousBackupsDescription": {
        "ContinuousBackupsStatus": "ENABLED",
        "PointInTimeRecoveryDescription": {
            "PointInTimeRecoveryStatus": "ENABLED",
            "RecoveryPeriodInDays": 35,
            "EarliestRestorableDateTime": "2026-07-17T16:59:48+09:00",
            "LatestRestorableDateTime": "2026-07-17T16:59:48+09:00"
        }
    }
}

$ ./bulk fill --table bulk-demo-202607 --generator default --numitems 1000000 --XWaitForDPU
Running action 'fill' with arguments: {'verb': 'fill', 'table': 'bulk-demo-202607', 'numitems': 1000000, 'generator': 'default', 'XWaitForDPU': True}

The bulk executor job cost consists of DynamoDB and Glue costs
For small jobs, the Glue cost portion is usually dominating
Using fewer Glue workers for small jobs, through the --XNumberOfWorkers parameter, will often reduce the Glue costs
For large jobs, where the cost is more significant, the DynamoDB cost portion is usually dominating
...

Starting Bulk Executor Glue Job...
Bulk Executor Glue Job started with job run ID: jr_cc8cd8a2f914848a358300957e2b509deb308f9111e1315a398af51205ef8f43
Job start time: 2026-07-17 17:00:14.777565
Press Ctrl+C to cancel Bulk Executor Glue Job.
Waiting 40 seconds for DPU metrics to gather...
Job completed successfully. Job duration: 0:05:48 (2.00 DPU hours)

Table: 'bulk-demo-202607'
Billing mode: On-demand

Table Item Count (approx): 0
Table Size (approx): 0 bytes

DynamoDB fill costs depend on how many items are being written and the size of the items.
Here we assume the command will insert 1,000,000 items
 with average size 161 bytes (based on peeking at generator output);
 each write incurs an average of 1 write units
Write units required to do such a fill (approx): 1,000,000
This does not include costs for secondary indexes!
Approx DynamoDB cost for On-demand writes consuming 1,000,000 WRUs (using ap-northeast-1 prices): $0.72

[bulk-demo-202607] Max write rate set to account quota limit: 40000
Total records filled: 1,000,000

$ ./bulk scancount --table bulk-demo-202607 --XWaitForDPU
Running action 'scancount' with arguments: {'verb': 'scancount', 'table': 'bulk-demo-202607', 'XWaitForDPU': True}
...
Starting Bulk Executor Glue Job...
Bulk Executor Glue Job started with job run ID: jr_1b026b88f39db20907cdef029f3625b7c13636772f2d391fc37ebbfe08c55aa8
Job start time: 2026-07-17 17:23:42.726485
Press Ctrl+C to cancel Bulk Executor Glue Job.
Waiting 40 seconds for DPU metrics to gather...
Job completed successfully. Job duration: 0:02:07 (0.87 DPU hours)

Table: 'bulk-demo-202607'
Billing mode: On-demand
...
[bulk-demo-202607] Max read rate set to account quota limit: 40000
Total records counted: 1,000,000

# スパースGSIをスキャンして特定条件を持つアイテム数だけカウント
$ ./bulk scancount --table orders --index i

# DynamoDBのFilterExpression構文で述語プッシュダウン
$ ./bulk scancount --table orders \
    --filter-expression "#touched > :touched" \
    --expression-names '{"#touched": "touched"}' \
    --expression-values '{":touched":1742359403.0}'

$ ./bulk count --table bulk-demo-202607 --where "payload IS NOT NULL" --XWaitForDPU
Running action 'count' with arguments: {'verb': 'count', 'table': 'bulk-demo-202607', 'where': 'payload IS NOT NULL', 'XWaitForDPU': True}
...
Starting Bulk Executor Glue Job...
Bulk Executor Glue Job started with job run ID: jr_8c2c15c98eb95c6fb59711dc2bfb595e77671a478e5c8d26caf06ad094816caa
Job start time: 2026-07-17 17:31:24.461771
...
Job completed successfully. Job duration: 0:01:38 (0.86 DPU hours)
...
[bulk-demo-202607] Max read rate set to account quota limit: 40000
Count of matching items: 333,300

$ ./bulk find --table bulk-demo-202607 --orderby sk --limit 5 --XWaitForDPU
Running action 'find' with arguments: {'verb': 'find', 'table': 'bulk-demo-202607', 'orderby': 'sk', 'limit': 5, 'XWaitForDPU': True}
...
Starting Bulk Executor Glue Job...
Bulk Executor Glue Job started with job run ID: jr_b659d776dcea832cfeaa103659b5790a8b3868da786b4235588632568bdf1847
Job start time: 2026-07-17 17:27:01.568679
...
Job completed successfully. Job duration: 0:01:44 (0.86 DPU hours)
...
5 matching items:
{"pk":"sykxuyok7xkn","sk":"0005jbh047i9zj3e2uuc3m3k"}
{"pk":"7pp6ktfvpniv","payload":"iecq6msn54xqiy6n2sefdrh4kitfdu1k42vv20u7l21cdc7kv8d726b3ohsyhq973b3fohzyn2yrmk93y221hrj6tiqu7cbzyl7un5l6vtfj61rkb1jyuhn3at03anotbdc38vfqd68ncn06y8d0qrqdd6i8db5fokvsjfdmpk09duuoq8q8zjh60xeslrrtq9ak654u","sk":"0007x9tp5kjt9mhgswpno5ah"}
{"pk":"shr4g0m6w5ve","#meta":"1hefbof557","sk":"000bcuqybiny543u75xk212d"}
{"pk":"t8u9672s629l","sk":"000d1r6ntki87dl4hg0s1fju"}
{"pk":"6hklcoiew3uw","payload":"nvhg53c54vn9aghwxn6c6ypwbfa2wcciz5cw4tasdbahcj82ts82llvlp3op6h4pxjete31wjm6pfeeoezra2ndko28tjqzie07uj7nv5mx1mayh7jlwe8a7o1a7wlg2lox8z3n7r44byw3p3h9dvu9kefd6s799gv23xtbcr20zxmyw21ugofhk9r368edpu7h9q58n","sk":"000eap94gmmyjdhf9g0tq3tf"}

Wrote 5 items in JSON format to s3://aws-glue-bulk-dynamodb-ap-northeast-1-123456789012-bn2aieu6r/output/jr_b659d776dcea832cfeaa103659b5790a8b3868da786b4235588632568bdf1847/

# タイムスタンプで新しい方から100件
$ ./bulk find --table bulk-demo-202607 --orderby timestamp desc --limit 100

# 複数属性でソートするときはクォートで1引数にまとめる
$ ./bulk find --table bulk-demo-202607 --orderby "pk asc, sk desc" --limit 100

# 演算子の前後の空白は必須。Spark SQLの構文が使える
$ ./bulk find --table bulk-demo-202607 --where "payload LIKE 'iecq%'" --limit 10

$ ./bulk update --table mytable --generator touched

$ ./bulk diff --table t --table2 t2
$ ./bulk diff --table t --table2 t2 --format full --s3
$ ./bulk diff --table t --table2 t2 --sample-fraction 0.1

$ ./bulk copy --source source --target target
$ ./bulk copy \
    --source arn:aws:dynamodb:us-east-1:123456789012:table/source \
    --target arn:aws:dynamodb:us-west-2:987654321098:table/target

$ ./bulk load-export --table target --s3-path "s3://<bucket-name>/prefix/AWSDynamoDB/01716790307109-5f9d6aaa" \
    --XMaxWriteRate 40000 --XTimeout 720
$ ./bulk revert-export --table target --s3-path "s3://<bucket-name>/prefix/AWSDynamoDB/01716790307109-5f9d6aaa"

$ ./bulk teardown
Running action 'teardown' with arguments: {'verb': 'teardown'}
Deleted 3 objects from aws-glue-bulk-dynamodb-ap-northeast-1-123456789012-bn2aieu6r
2026-07-17 18:19:32,554 WARNING [MainThread] root - Bucket aws-glue-bulk-dynamodb-ap-northeast-1-123456789012-bn2aieu6r is not empty and could not be deleted. Continuing with teardown.
Detaching managed policy 'AWSGlueServiceRole'...
Detaching managed policy 'AmazonDynamoDBFullAccess'...
Deleting inline policy 'MinimalPricingAccess'...
Deleting inline policy 'MinimalQuotasAccess'...
Bulk Executor Glue Job Role deleted: AWSGlueServiceRoleBulkDynamoDB-DdbReadWrite-ap-northeast-1
Bulk Executor Glue Job deleted successfully.
Deleted Glue connection 'bulk-dynamodb-connection'.

    Bulk Executor environment configurations:

      AWS -
        Account: 123456789012
        Region: ap-northeast-1
        Role: arn:aws:sts::123456789012:assumed-role/xxxxxxx/botocore-session-xxxxxxxxxx

$ aws s3 rm s3://aws-glue-bulk-dynamodb-ap-northeast-1-123456789012-bn2aieu6r --recursive
delete: s3://aws-glue-bulk-dynamodb-ap-northeast-1-123456789012-bn2aieu6r/output/jr_.../part-00000-....json

$ aws s3 rb s3://aws-glue-bulk-dynamodb-ap-northeast-1-123456789012-bn2aieu6r
remove_bucket: aws-glue-bulk-dynamodb-ap-northeast-1-123456789012-bn2aieu6r

$ aws dynamodb delete-table --table-name bulk-demo-202607 --region ap-northeast-1

$ aws glue get-job --job-name bulk_dynamodb --region ap-northeast-1
An error occurred (EntityNotFoundException) when calling the GetJob operation: Job not found.

$ aws s3 ls | grep aws-glue-bulk-dynamodb
$ aws iam list-roles \
    --query 'Roles[?starts_with(RoleName,`AWSGlueServiceRoleBulkDynamoDB`)].RoleName' \
    --output text