[Update] 19 operators have been added to MWAA Serverless, enabling workflows to trigger other workflows.
This page has been translated by machine translation. View original
Hello. This is Takeda from the Service Development Division.
Amazon SageMaker Unified Studio Workflows now supports new operators. The targets are Amazon Bedrock, Amazon S3 Tables, Amazon S3 Vectors, and AWS Glue Data Catalog.
These operators can be used with MWAA Serverless. They can be specified either from the Unified Studio visual creator or from YAML definitions.
Among the additions, there was an operator for triggering workflows, so I actually ran it to verify.
Breakdown of the 19 Added Operators
The additions are listed in the MWAA Serverless supported operators list.
The announcement mentions "19 new operators," but does not enumerate which operators they are. The 5 services listed in the announcement are Bedrock guardrail, S3 Tables, S3 Vectors, Glue Data Catalog, and MWAA Serverless. Counting the applicable ones from this list gives 19.
| Category | Operator | Count |
|---|---|---|
| Workflows | MwaaServerlessCreateWorkflowOperatorMwaaServerlessStartWorkflowRunOperator |
2 |
| Catalog | GlueCatalogCreateDatabaseOperatorGlueCatalogCreateTableOperatorGlueCatalogDeleteDatabaseOperatorGlueCatalogDeleteTableOperator |
4 |
| Machine learning | BedrockCreateGuardrailOperatorBedrockCreateGuardrailVersionOperatorBedrockDeleteGuardrailOperator |
3 |
| Storage (S3 Tables) | S3TablesCreateTableBucketOperatorS3TablesCreateNamespaceOperatorS3TablesCreateTableOperatorS3TablesDeleteTableBucketOperatorS3TablesDeleteNamespaceOperatorS3TablesDeleteTableOperator |
6 |
| Storage (S3 Vectors) | S3VectorsCreateVectorBucketOperatorS3VectorsCreateIndexOperatorS3VectorsDeleteVectorBucketOperatorS3VectorsDeleteIndexOperator |
4 |
The total is 19. AWS did not explicitly state "these 19," so this is derived from matching the categories and counts.
For Bedrock, operators such as BedrockInvokeModelOperator and BedrockCreateKnowledgeBaseOperator have existed for some time. The 3 added this time are guardrail-related. The announcement body also states "managing guardrails."
All of these operators are already implemented in the open-source Amazon provider for Apache Airflow.
With this update, they can now be called from MWAA Serverless.
Triggering Another Workflow from a Workflow
I tried the 2 operators in the Workflows category from the additions.
MwaaServerlessCreateWorkflowOperator- Creates a workflowMwaaServerlessStartWorkflowRunOperator- Starts a workflow run
MwaaServerlessStartWorkflowRunOperator is an operator for triggering another workflow from within a workflow. It can be used similarly to Airflow's TriggerDagRunOperator in MWAA Serverless.
Previously, when I verified MWAA Serverless EventBridge support, I wrote the following about workflow chaining.
MWAA Serverless has limited means to wait for the completion of other workflows, and waiting with a sensor incurs task execution time charges during the wait. Being able to set up a chain of Succeeded event → Lambda →
StartWorkflowRunis personally a welcome point.
At that time, EventBridge and Lambda were involved in the chaining. With MwaaServerlessStartWorkflowRunOperator, you can trigger a downstream workflow simply by placing a task at the end of the upstream workflow. I verified that this actually works.
Verification Setup
I prepared two workflows.
- Downstream workflow: Writes one marker file to S3 (
S3CreateObjectOperator) - Upstream workflow: After writing a marker to S3, triggers the downstream using
MwaaServerlessStartWorkflowRunOperator
The downstream YAML definition is as follows.
downstream_wf:
dag_id: downstream_wf
schedule: null
default_args:
owner: airflow
start_date: "2024-01-01"
tasks:
write_marker:
operator: airflow.providers.amazon.aws.operators.s3.S3CreateObjectOperator
task_id: write_marker
s3_bucket: amzn-s3-demo-bucket
s3_key: markers/downstream_ran.txt
data: "downstream executed"
replace: true
The upstream YAML definition is as follows. The trigger_downstream task specifies the ARN of the downstream workflow.
upstream_wf:
dag_id: upstream_wf
schedule: null
default_args:
owner: airflow
start_date: "2024-01-01"
tasks:
write_upstream_marker:
operator: airflow.providers.amazon.aws.operators.s3.S3CreateObjectOperator
task_id: write_upstream_marker
s3_bucket: amzn-s3-demo-bucket
s3_key: markers/upstream_ran.txt
data: "upstream executed"
replace: true
trigger_downstream:
operator: airflow.providers.amazon.aws.operators.mwaa_serverless.MwaaServerlessStartWorkflowRunOperator
task_id: trigger_downstream
workflow_arn: arn:aws:airflow-serverless:ap-northeast-1:123456789012:workflow/downstream-wf-xxxxxxxxxx
dependencies:
- write_upstream_marker
The required parameter for MwaaServerlessStartWorkflowRunOperator is workflow_arn. Additionally, override_parameters and workflow_version can be specified.
Execution Role
MWAA Serverless has one execution role per workflow. The trust policy specifies airflow-serverless.amazonaws.com.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "airflow-serverless.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
For the upstream execution role, in addition to S3 write and CloudWatch Logs, I added airflow-serverless:StartWorkflowRun. This permission is what triggers the downstream.
{
"Effect": "Allow",
"Action": ["airflow-serverless:StartWorkflowRun"],
"Resource": "*"
}
Results
To create workflows, place the YAML in S3 and then run create-workflow. The workflow_arn written in the upstream YAML is not determined until the downstream is created, so create the downstream first.
First, create the downstream.
aws mwaa-serverless create-workflow \
--name downstream-wf \
--definition-s3-location '{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"definitions/downstream.yaml"}' \
--role-arn arn:aws:iam::123456789012:role/mwaa-sls-exec
The response returns the downstream ARN.
{
"WorkflowArn": "arn:aws:airflow-serverless:ap-northeast-1:123456789012:workflow/downstream-wf-xxxxxxxxxx"
}
Enter this ARN into the workflow_arn of the trigger_downstream task in the upstream YAML mentioned earlier. After editing, create the upstream.
aws mwaa-serverless create-workflow \
--name upstream-wf \
--definition-s3-location '{"Bucket":"amzn-s3-demo-bucket","ObjectKey":"definitions/upstream.yaml"}' \
--role-arn arn:aws:iam::123456789012:role/mwaa-sls-exec
Then run the upstream.
aws mwaa-serverless start-workflow-run \
--workflow-arn arn:aws:airflow-serverless:ap-northeast-1:123456789012:workflow/upstream-wf-xxxxxxxxxx
The upstream run ended with SUCCESS, and both tasks write_upstream_marker and trigger_downstream completed. Looking at the downstream side, one new run had appeared after the upstream execution time, with a status of SUCCESS and RunType of ON_DEMAND. The downstream marker file was also written. The upstream's trigger_downstream task successfully triggered the downstream.
The triggered downstream run is recorded as ON_DEMAND, the same as a manual run from the console or CLI. No field was found to distinguish a run triggered via an operator.
What Happens When the Permission Is Removed
The permissions required for this operator are not documented. I removed only airflow-serverless:StartWorkflowRun from the upstream execution role and ran it again.
The result was FAILED. Looking at the individual tasks, the S3CreateObjectOperator task was SUCCESS. Only the MwaaServerlessStartWorkflowRunOperator task was FAILED. No new run had been added on the downstream side, and the downstream marker was not written either.
{
"TaskInstanceId": "..._write_upstream_marker_1",
"Status": "SUCCESS",
"OperatorName": "S3CreateObjectOperator"
},
{
"TaskInstanceId": "..._trigger_downstream_1",
"Status": "FAILED",
"OperatorName": "MwaaServerlessStartWorkflowRunOperator"
}
list-task-instances returns OperatorName and Status per task, so it is possible to trace which operator failed. However, the reason for the failure does not appear here, so the cause must be checked in the task logs.
Waiting for Completion Is Not Yet Possible
While triggering is now possible, MwaaServerlessWorkflowRunSensor for waiting for run completion is not in the supported list. Since MWAA Serverless only accepts operators from the list, even if you trigger a downstream, it is currently not possible to wait for its completion before proceeding to the next task. The situation of "limited means to wait for the completion of other workflows" mentioned earlier remains: while the means of triggering have increased, the issue of waiting for completion persists.
If you want to detect completion, you will still need to use events delivered to EventBridge as before.
Summary
19 more operators are now available in MWAA Serverless.
- The additions are in the MWAA Serverless supported operators list. They can also be used from YAML definitions without using Unified Studio
- The breakdown is 2 for MWAA Serverless, 4 for Glue Data Catalog, 3 for Bedrock guardrail, 6 for S3 Tables, and 4 for S3 Vectors
MwaaServerlessStartWorkflowRunOperatorallows triggering another workflow from within a workflow. A setup where upstream triggers downstream worked successfully. Chaining can be set up without involving EventBridge and Lambda- Triggering the downstream requires
airflow-serverless:StartWorkflowRunon the upstream execution role. Removing it caused the trigger task to fail MwaaServerlessWorkflowRunSensorfor waiting for completion is not in the supported list, and waiting for completion looks likely to remain a challenge going forward
I hope this is helpful to someone.
