[Update] PythonOperator and BashOperator are now available in Amazon MWAA Serverless

Hello. I'm Takeda from the Service Development Department.

Amazon MWAA Serverless now supports PythonOperator and BashOperator.

https://aws.amazon.com/about-aws/whats-new/2026/08/mwaa-serverless-pythonoperator-bashoperator/

Until now, MWAA Serverless only supported operators for AWS services listed in the supported list. With this update, you can upload code to S3 and pass it via create-workflow/update-workflow to run your own Python functions and shell scripts on workers.

There is an interesting sentence in the announcement:

The service snapshots your code at workflow creation time and uses that snapshot for all subsequent runs, ensuring consistency across executions.

It states that the code is snapshotted at workflow creation time, and subsequent executions use that snapshot. Replacing the code in S3 will not be reflected. I was curious about how this affects operations, so I verified how S3 code and workflow versions correspond to each other.

The environment at the time of verification was Airflow 3.0.6 / apache-airflow-providers-standard 1.6.0.

How to Use

Code is passed separately from the workflow definition (YAML) via the Code parameter. It can be a single .py file, a single .sh file, or a .zip containing multiple files.

https://docs.aws.amazon.com/mwaa/latest/mwaa-serverless-userguide/operators-python-bash-detail.html

The code used this time consists of 2 files. I prepared a function that returns a VERSION constant, and used whether it returns v1 or v2 to identify which code is running.

# mymod.py
import helper

VERSION = "v1"


def version():
    print(f"code version = {VERSION}")
    return VERSION

# helper.py
def greet(name):
    return f"hello, {name}"

For the zip, place files directly under the root. Archiving them inside a directory will cause a No module named 'mymod' error at runtime (the create itself will succeed).

zip pkg-v1.zip mymod.py helper.py

The definition is as follows. python_callable is written as a string in the format module_name.function_name. The return value of py_version is stored in XCom, so it is retrieved with xcom_pull and printed in the subsequent bash_show.

pybash_snapshot:
  dag_id: pybash_snapshot
  schedule: null
  default_args:
    owner: airflow
    start_date: "2024-01-01"
  tasks:
    py_version:
      operator: airflow.providers.standard.operators.python.PythonOperator
      task_id: py_version
      python_callable: mymod.version
    bash_show:
      operator: airflow.providers.standard.operators.bash.BashOperator
      task_id: bash_show
      bash_command: "echo \"xcom={{ ti.xcom_pull(task_ids='py_version') }}\""
      dependencies:
        - py_version

Place the definition and zip in S3, then create it with create-workflow using --code. The bucket has versioning enabled.

aws mwaa-serverless create-workflow \
  --name pybash-snapshot \
  --definition-s3-location '{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"definitions/snapshot.yaml"}' \
  --code '{"S3Location":{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"code/pkg.zip"}}' \
  --role-arn arn:aws:iam::123456789012:role/mwaa-pybash-test-role

The execution role is granted s3:GetObject/s3:GetObjectVersion/s3:ListBucket on the definition and code buckets, as well as write access to CloudWatch Logs.

Looking at get-workflow, it returns Code and CodeSnapshottedAt.

{
    "WorkflowVersion": "d495e078...",
    "Code": {
        "S3Location": {
            "Bucket": "amzn-s3-demo-bucket",
            "ObjectKey": "code/pkg.zip"
        }
    },
    "CodeSnapshottedAt": "2026-08-25T07:17:05.111992+00:00"
}

The documentation states that VersionId is included in Code.S3Location. However, when creating without specifying VersionId, it was not returned even on a versioning-enabled bucket (as of August 25, 2026). It was only returned when VersionId was explicitly specified in an update.

After execution, the result is SUCCESS, and the return value of py_version is stored in Xcom of get-task-instance.

{
    "TaskId": "py_version",
    "Status": "SUCCESS",
    "OperatorName": "PythonOperator",
    "Xcom": {
        "return_value": "\"v1\""
    }
}

The bash_show logs also showed xcom=v1.

The Difference Between Overwriting S3 and Calling update

There are two types of versions involved. The S3 VersionId is the version of the S3 object, which determines which code is captured at create/update time. WorkflowVersion is assigned by MWAA Serverless each time create/update is called, and determines which definition and snapshotted code is used at execution time.

Overwriting S3 Does Not Change the Code That Gets Executed

Upload a zip with VERSION = "v2" to the same key code/pkg.zip, overwriting the existing one. Do not call update-workflow.

aws s3api put-object --bucket amzn-s3-demo-bucket --key code/pkg.zip --body pkg-v2.zip

Even after overwriting, get-workflow shows no change in WorkflowVersion or CodeSnapshottedAt. Running the workflow in this state still returned v1.

{
    "Xcom": {
        "return_value": "\"v1\""
    }
}

Just as the announcement states. Simply replacing the file in S3 does not change anything on the workflow side.

Calling update Creates a New WorkflowVersion

Call update-workflow specifying the same key in --code (omitting VersionId).

aws mwaa-serverless update-workflow \
  --workflow-arn arn:aws:airflow-serverless:ap-northeast-1:123456789012:workflow/pybash-snapshot-xxxxxxxxxx \
  --definition-s3-location '{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"definitions/snapshot.yaml"}' \
  --code '{"S3Location":{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"code/pkg.zip"}}' \
  --role-arn arn:aws:iam::123456789012:role/mwaa-pybash-test-role

The response returns a new WorkflowVersion, and CodeSnapshottedAt is updated to the time of the update. Looking at list-workflow-versions, two versions are listed.

{
    "WorkflowVersions": [
        {
            "WorkflowVersion": "953984f3...",
            "IsLatestVersion": true,
            "CreatedAt": "2026-08-25T08:27:26.984000+00:00"
        },
        {
            "WorkflowVersion": "d495e078...",
            "IsLatestVersion": false,
            "CreatedAt": "2026-08-25T07:17:04.707084+00:00"
        }
    ]
}

The new version has IsLatestVersion: true. When --workflow-version is omitted in start-workflow-run, the version with this flag set to true is executed. Running it returned v2. An update without VersionId captures the latest object currently in S3.

Each element of list-workflow-versions does not include Code. There is no way to trace after the fact via the API which WorkflowVersion was created from which S3 object.

Running with Old Code

To run the old code just once, specify the old version with --workflow-version in start-workflow-run.

aws mwaa-serverless start-workflow-run \
  --workflow-arn arn:aws:airflow-serverless:ap-northeast-1:123456789012:workflow/pybash-snapshot-xxxxxxxxxx \
  --workflow-version d495e078...

The result was v1.

If you also want the default execution (without specifying a version) to use the old code, call update with the old VersionId explicitly specified in --code. The latest S3 object remains v2.

aws mwaa-serverless update-workflow \
  --workflow-arn arn:aws:airflow-serverless:ap-northeast-1:123456789012:workflow/pybash-snapshot-xxxxxxxxxx \
  --definition-s3-location '{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"definitions/snapshot.yaml"}' \
  --code '{"S3Location":{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"code/pkg.zip","VersionId":"G5eRYx6s..."}}' \
  --role-arn arn:aws:iam::123456789012:role/mwaa-pybash-test-role

A third WorkflowVersion is created, and this time VersionId is included in Code.S3Location of get-workflow.

{
    "Code": {
        "S3Location": {
            "Bucket": "amzn-s3-demo-bucket",
            "ObjectKey": "code/pkg.zip",
            "VersionId": "G5eRYx6s..."
        }
    },
    "CodeSnapshottedAt": "2026-08-25T08:44:20.406374+00:00"
}

Running it returns v1. Even though the latest S3 object is v2, the explicitly specified version was captured. The old version did not become the latest again; rather, a third version containing the old code was created, and IsLatestVersion was assigned to it. The default execution also returns v1.

Can the Workflow Still Run After Deleting the Code from S3?

To verify whether the snapshot references S3 at execution time, I deleted all versions of code/pkg.zip.

# Run for each of the 3 VersionIds
aws s3api delete-object \
  --bucket amzn-s3-demo-bucket \
  --key code/pkg.zip \
  --version-id "$VERSION_ID"

After deletion, both Versions and DeleteMarkers in list-object-versions become empty. head-object returns 404, and get-object with a VersionId returns NoSuchVersion.

In this state, I ran each of the three WorkflowVersions by specifying --workflow-version.

All succeeded. At least these three WorkflowVersions do not reference the original S3 objects at execution time. It can be assumed that an executable snapshot is retained in the service's internal management area (the internal storage format is not publicly disclosed). However, I have not verified the retention period of WorkflowVersions or how long they remain executable after the original is deleted. It is safer to keep the original objects in S3 as well.

I tried the same thing with a bucket that had versioning disabled. Even after overwriting the same key after create, the result remained v1, and execution was still possible after deleting the object. S3 versioning is not a prerequisite for snapshots. On the other hand, VersionId is needed to explicitly capture a specific version, so versioning is necessary in that sense. I understand that the documentation recommends versioning in production for this reason.

Two Points to Note When Updating the Definition

What I felt required care in operations was the behavior when modifying the definition.

Omitting --code Does Not Carry Over the Code

I tested what happens when calling update-workflow without --code when only the definition file needs to be updated. I called it with only --definition-s3-location and --role-arn.

aws mwaa-serverless update-workflow \
  --workflow-arn arn:aws:airflow-serverless:ap-northeast-1:123456789012:workflow/pybash-nocode-xxxxxxxxxx \
  --definition-s3-location '{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"definitions/nocode-update-v2def.yaml"}' \
  --role-arn arn:aws:iam::123456789012:role/mwaa-pybash-test-role

The update succeeds. However, looking at get-workflow, Code and CodeSnapshottedAt had disappeared from the response. Running it caused a No module named 'mymod' error during DAG generation, resulting in py_version being FAILED and downstream tasks being UPSTREAM_FAILED.

Running the immediately preceding WorkflowVersion by specifying --workflow-version succeeded with v1. The code was not removed from the entire workflow; rather, it is a state where "the newly created version has no code." At least for Code, the previous value is not carried over on update. Even when only fixing the definition, always include --code.

Re-specifying the Same Key Captures the Code Currently in S3

So is it safe as long as --code is included? As long as VersionId is omitted, the latest object currently in S3 will be captured.

During this verification, I ran update twice on a separate workflow to fix a typo in the definition. In between, I had overwritten the same key code/pkg.zip with v2 for the snapshot verification, so on the second update, the code changed from v1 to v2. Even if you think you only touched the definition, the code can change.

To summarize:

• Omitting --code → Code is not carried over to the new version
• Re-specifying only the same key in --code → The latest code currently in S3 is captured

The countermeasure is to always explicitly specify the VersionId you want to capture in --code, or to use a unique ObjectKey per release that is never overwritten.

This is especially easy to get caught by when multiple workflows share the same code key, or when code uploads and definition fixes are done by different people or different pipelines. Since the documentation allows specifying VersionId for --definition-s3-location on the definition side as well, pinning both the definition and the code seems advisable for reproducible deployments.

What to Record in Release Notes

Since the API does not allow you to trace "which WorkflowVersion corresponds to which code" after the fact, you should keep your own records at update time. For example, a correspondence table like the following:

Since the update-workflow response does not include the input VersionId, keep it together with the command input.

Notes Specific to Code Packages

There were three issues I encountered while running things.

bash_command Ending in .sh Is Treated as a Template File

When I uploaded a single .sh as code and called it with bash_command: "bash run.sh", the following error occurred:

TemplateNotFound: 'bash run.sh' not found in search paths: '/usr/local/airflow/dags'

Due to Airflow's behavior, a bash_command ending in .sh or .bash is resolved as a Jinja template file. This is the same as with conventional MWAA, but the opportunity to run into it is likely to increase now that .sh files can be passed as code.

For Python DAGs in Airflow 2.8 and later, the current approach is to wrap it in literal() to disable rendering entirely. In Airflow 3, it can be imported with from airflow.sdk import literal. However, since MWAA Serverless definitions are in YAML and bash_command only accepts a string, this method cannot be used. As an old-fashioned workaround, appending a trailing space like "bash run.sh " causes it to be executed as a plain string and succeeds.

The Working Directory of BashOperator Varies Depending on the Specified Operator Path

The documentation states that "BashOperator scripts are executed with /usr/local/airflow/dags as the working directory." When I ran the same pwd command with only the operator specification changed, the results differed.

The providers path version runs in a temporary directory, which is the default for Airflow's BashOperator. Only the old path airflow.operators.bash.BashOperator used the dags directory. I'm not sure why there is a difference.

The approach of "calling a script by name" like bash run.sh worked with both. This is because PATH includes /usr/local/airflow/dags, and bash looks up the file from PATH using the argument. For scripts that read and write files using relative paths, it is safer to explicitly specify cwd.

Actually, at first I also had bash_show output grep -n 'VERSION =' mymod.py. Since I was using the providers path, this grep was failing with No such file or directory. I had proceeded with the verification by only looking at the XCom value, and only noticed when I reviewed the logs.

Pre-installed Packages Take Precedence Over Bundled Ones

The documentation states that "even if you bundle a different version of a pre-installed package, the pre-installed version will be used." I confirmed this by bundling lz4 (pre-installed version is 4.4.4) as version 4.3.3.

importlib.metadata.version("lz4") → 4.3.3
lz4.__version__                   → 4.4.4
lz4.__file__                      → /opt/mwaa-serverless-venv/lib64/python3.12/site-packages/lz4/__init__.py

importlib.metadata.version() and pip show indicated the bundled 4.3.3. However, the imported module is the pre-installed 4.4.4, as shown by __file__ and __version__. This is as documented, but the distribution metadata and the actual import source are inconsistent. When verifying the version at runtime, check __file__ and the package's own version attribute as well.

Looking at pip list, there were more pre-installed packages than listed in the documentation, including tabulate, requests, and redshift-connector. Checking beforehand can help avoid unnecessary zip bloat.

Other Things I Noticed

• The return value of python_callable is stored in XCom and appears as a JSON string in Xcom.return_value of get-task-instance. BashOperator also stores the last line as return_value
• op_kwargs can be passed from YAML. The combination of params and {{ params.xxx }} also works and can be overridden with start-workflow-run --override-parameters. On the other hand, --override-parameters values were not available in the task's environment variables
• {{ ds }} worked. On the other hand, writing {{ run_id }} in bash_command caused DAG template rendering failed: 'run_id' is undefined and the execution failed before running. As with the case where ti.run_id could not be referenced in a previous article, the variables available in templates are restricted
• In a configuration without VPC, boto3 was able to reach S3, but sts get-caller-identity resulted in a ConnectTimeoutError
• BranchPythonOperator is rejected at create time with is not supported. PythonVirtualenvOperator and PythonSensor are rejected because python_callable is treated as a parameter exclusive to PythonOperator. The only two operators available for custom code are PythonOperator and BashOperator

Summary

PythonOperator and BashOperator are now available in MWAA Serverless. Place code in S3 and pass it with --code.

Organizing the snapshot behavior by operation:

Even after deleting the original objects from S3, the three WorkflowVersions I verified were still executable. On the other hand, VersionId in get-workflow is only returned when explicitly specified, and list-workflow-versions does not include Code. You will need to keep your own records of which WorkflowVersion corresponds to which code, using S3 VersionId and the zip hash.

More than the ability to pass code itself, recording the correspondence between workflow versions and code will likely be the more important concern in operations.

I hope this is helpful to someone.