I tried building a FastQC execution environment using AWS Batch with only the AWS Management Console

Introduction

Hello, I'm Horiguchi.

What kind of environment do you usually use for RNA-seq analysis?
I think many of you may be doing it on a lab server or personal PC.

This time, I will introduce step by step how to build a FastQC execution environment on AWS using the AWS Management Console. As a minimal configuration, I would like to create an S3 bucket as a storage destination for data and run just one FastQC job with AWS Batch.

I hope this will be helpful as a reference for what kind of configuration would be used when running analyses done on a lab server or personal PC on the cloud.

Notes

• This article uses public data (GATK Test Data)
• This article does not provide scientific interpretation of analysis results

Terminology

RNA-seq
This is a method to investigate which genes are being used and how much in a cell. When a gene is used, a molecule called RNA, which copies that information, is produced. In RNA-seq, the sequence of RNA is read and its quantity is measured to estimate how active genes are.

FASTQ
Sequences read by a device called a sequencer are generally saved in files in a format called FASTQ. FASTQ files record the sequences that were read and their read quality.

FastQC
This is software for checking whether there are any quality problems in the sequence data saved in FASTQ files, using graphs and tables.

In other words, to summarize the purpose of this article in one sentence:
"Let's check the quality of sequence data read by RNA-seq on AWS."

Thank you for reading.

Configuration to Create This Time

Overall architecture to be created in this article

Since we aim to complete everything in the Management Console this time, the FastQC container image will not be built on a local PC either, but will be created with CodeBuild and saved to ECR.

The main AWS services used this time are as follows.

Note that Fargate is used for the AWS Batch Compute environment.
There is no need to create or manage EC2 instances yourself.

1. Upload Input FASTQ to S3

Let's get started right away.

First, create an S3 bucket to store input data and FastQC output.
Open the Amazon S3 console and select "Create bucket."

※ Traditionally, bucket names needed to be unique across all AWS accounts, but using account regional namespaces, the same base name can be used in different AWS accounts or regions. However, it must be unique within the same AWS account and same region.

Leave the settings basically at their defaults, and keep "Block all public access" enabled.

Creating an S3 bucket

Next, create the following 2 prefixes (folders in the console) inside the bucket.

• input_data/: Store the FASTQ files to be analyzed
• output_data/: Store FastQC results

Create input_data and output_data folders inside the S3 bucket

For paired-end data, the input data is arranged in a structure like the following.

input_data/
├── sample1_R1.fastq.gz
└── sample1_R2.fastq.gz

On the other hand, for single-end data, there will be only one FASTQ file.

Note down the S3 URI of the uploaded object, as it will be used later when creating the AWS Batch job definition.

2. Create an ECR Repository for FastQC

2-1. Prerequisites

In AWS Batch, jobs are executed using container images that bundle together the necessary software and execution procedures, rather than installing analysis software directly on the host environment. Therefore, it is first necessary to create a container image that incorporates FastQC, Java, AWS CLI, and other tools.

A container image is an execution environment that bundles together the software and settings needed for analysis. By fixing the FastQC version and required libraries within the image, there is no need to build the environment each time it runs, and analysis can be executed under the same conditions.

This time, since we aim to complete everything using the AWS Management Console, we will use AWS CodeBuild to create the container image for FastQC without operating Docker on a local PC. The created image is saved to Amazon ECR, and when running a job with AWS Batch, Fargate retrieves it from ECR.

2-2. Creating the ECR Repository

Now let's actually create the ECR repository to store the FastQC container image.

From Amazon ECR's "Create repository," create a private repository.
The repository name for this time is as follows.

rna-seq/fastq-test

Since the repository name can include /, the project name and purpose can be expressed hierarchically.

ECR repository creation screen

After creation, note down the ECR repository URI.

AWSAccountID.dkr.ecr.us-east-1.amazonaws.com/rna-seq/fastq-test

This URI will be used later in the AWS Batch job definition.

3. Create the FastQC Container with CodeBuild

Create the CodeBuild project with the following settings.

Basic settings of the CodeBuild project

Environment settings of the CodeBuild project

Enable Privileged Mode

In the CodeBuild environment settings, enable privileged mode.
Expand "Additional configuration" at the bottom of the "Environment" section and check the box for privilege grant.

CodeBuild privilege grant settings

What is Privileged Mode?

The CodeBuild build environment itself also runs in a container. This time, the following commands are further executed within it.

docker build
docker push

Privileged mode needs to be enabled to execute Docker image manipulation commands such as docker build inside CodeBuild.

Note that privileged mode and IAM permissions are separate things.

Required to execute Docker build and other commands
 → Privileged mode
Required to push to ECR
 → IAM policy of the CodeBuild service role

Only when both privileged mode and appropriate IAM policies are in place can the created Docker image be pushed to ECR.

Setting Environment Variables

The following environment variables were set for the CodeBuild project.

CodeBuild environment variables

For ECR_REPOSITORY, set only the repository name, not the entire ECR URI.
Since FastQC 0.12.1 is used this time, the image tag is set to 0.12.1.

Processing Executed at Container Startup

The container created this time performs the following processing when started.

• Download FASTQ from S3
• Run FastQC
• Upload HTML and ZIP to S3

This time, since the Dockerfile and shell script contents are entered as a single line in the Build commands field, each was encoded in Base64 format. During the build, they are decoded with base64 -d and output as files.

The commands set in the Build commands field are as follows.

echo RlJPTSBwdWJsaWMuZWNyLmF3cy9hbWF6b25saW51eC9hbWF6b25saW51eDoyMDIzCkFSRyBGQVNUUUNfVkVSU0lPTj0wLjEyLjEKUlVOIGRuZiBpbnN0YWxsIC15IGphdmEtMTctYW1hem9uLWNvcnJldHRvLWhlYWRsZXNzIHBlcmwgdW56aXAgd2dldCBhd3NjbGkgZ3ppcCB0YXIgJiYgZG5mIGNsZWFuIGFsbApSVU4gd2dldCAtcSAiaHR0cHM6Ly93d3cuYmlvaW5mb3JtYXRpY3MuYmFicmFoYW0uYWMudWsvcHJvamVjdHMvZmFzdHFjL2Zhc3RxY192JHtGQVNUUUNfVkVSU0lPTn0uemlwIiAtTyAvdG1wL2Zhc3RxYy56aXAgJiYgdW56aXAgL3RtcC9mYXN0cWMuemlwIC1kIC9vcHQgJiYgY2htb2QgK3ggL29wdC9GYXN0UUMvZmFzdHFjICYmIGxuIC1zIC9vcHQvRmFzdFFDL2Zhc3RxYyAvdXNyL2xvY2FsL2Jpbi9mYXN0cWMgJiYgcm0gL3RtcC9mYXN0cWMuemlwCkNPUFkgcnVuX2Zhc3RxYy5zaCAvdXNyL2xvY2FsL2Jpbi9ydW5fZmFzdHFjLnNoClJVTiBjaG1vZCAreCAvdXNyL2xvY2FsL2Jpbi9ydW5fZmFzdHFjLnNoCkVOVFJZUE9JTlQgWyIvdXNyL2xvY2FsL2Jpbi9ydW5fZmFzdHFjLnNoIl0K | base64 -d > Dockerfile && echo IyEvdXNyL2Jpbi9lbnYgYmFzaApzZXQgLWV1byBwaXBlZmFpbAoKOiAiJHtJTlBVVF9SMTo/SU5QVVRfUjEgaXMgcmVxdWlyZWR9Igo6ICIke09VVFBVVF9TMzo/T1VUUFVUX1MzIGlzIHJlcXVpcmVkfSIKSU5QVVRfUjI9IiR7SU5QVVRfUjI6LU5PTkV9IgpUSFJFQURTPSIke1RIUkVBRFM6LTJ9IgoKbWtkaXIgLXAgL3dvcmsvaW5wdXQgL3dvcmsvb3V0cHV0CgpSMV9MT0NBTD0iL3dvcmsvaW5wdXQvJChiYXNlbmFtZSAiJElOUFVUX1IxIikiCmF3cyBzMyBjcCAiJElOUFVUX1IxIiAiJFIxX0xPQ0FMIgpJTlBVVFM9KCIkUjFfTE9DQUwiKQoKaWYgW1sgIiRJTlBVVF9SMiIgIT0gIk5PTkUiICYmIC1uICIkSU5QVVRfUjIiIF1dOyB0aGVuCiAgUjJfTE9DQUw9Ii93b3JrL2lucHV0LyQoYmFzZW5hbWUgIiRJTlBVVF9SMiIpIgogIGF3cyBzMyBjcCAiJElOUFVUX1IyIiAiJFIyX0xPQ0FMIgogIElOUFVUUys9KCIkUjJfTE9DQUwiKQpmaQoKZmFzdHFjIC0tdGhyZWFkcyAiJFRIUkVBRFMiIC0tb3V0ZGlyIC93b3JrL291dHB1dCAiJHtJTlBVVFNbQF19Igphd3MgczMgY3AgL3dvcmsvb3V0cHV0LyAiJHtPVVRQVVRfUzMlL30vIiAtLXJlY3Vyc2l2ZQo= | base64 -d > run_fastqc.sh && ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) && REGION=$AWS_DEFAULT_REGION && ECR_URI=${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${ECR_REPOSITORY} && aws ecr get-login-password --region ${REGION} | docker login --username AWS --password-stdin ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com && docker build -t ${ECR_URI}:${IMAGE_TAG} . && docker push ${ECR_URI}:${IMAGE_TAG}

The content of the Dockerfile before encoding is as follows.

FROM public.ecr.aws/amazonlinux/amazonlinux:2023

ARG FASTQC_VERSION=0.12.1

RUN dnf install -y \
      java-17-amazon-corretto-headless \
      perl \
      unzip \
      wget \
      awscli \
      gzip \
      tar \
    && dnf clean all

RUN wget -q \
      "https://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v${FASTQC_VERSION}.zip" \
      -O /tmp/fastqc.zip \
    && unzip /tmp/fastqc.zip -d /opt \
    && chmod +x /opt/FastQC/fastqc \
    && ln -s /opt/FastQC/fastqc /usr/local/bin/fastqc \
    && rm /tmp/fastqc.zip

COPY run_fastqc.sh /usr/local/bin/run_fastqc.sh

RUN chmod +x /usr/local/bin/run_fastqc.sh

ENTRYPOINT ["/usr/local/bin/run_fastqc.sh"]

The shell script that executes FastQC before encoding is as follows.

#!/usr/bin/env bash
set -euo pipefail

: "${INPUT_R1:?INPUT_R1 is required}"
: "${OUTPUT_S3:?OUTPUT_S3 is required}"

INPUT_R2="${INPUT_R2:-NONE}"
THREADS="${THREADS:-2}"

mkdir -p /work/input /work/output

R1_LOCAL="/work/input/$(basename "$INPUT_R1")"
aws s3 cp "$INPUT_R1" "$R1_LOCAL"
INPUTS=("$R1_LOCAL")

if [[ "$INPUT_R2" != "NONE" && -n "$INPUT_R2" ]]; then
  R2_LOCAL="/work/input/$(basename "$INPUT_R2")"
  aws s3 cp "$INPUT_R2" "$R2_LOCAL"
  INPUTS+=("$R2_LOCAL")
fi

fastqc \
  --threads "$THREADS" \
  --outdir /work/output \
  "${INPUTS[@]}"

aws s3 cp \
  /work/output/ \
  "${OUTPUT_S3%/}/" \
  --recursive

This time, since a CodeBuild project with "No source" was used, commands to generate the Dockerfile and shell script and then build and push the Docker image were set in the build commands field.

CodeBuild build command settings

Supplement: Creating a Container Image in a Local Environment

This time, we used CodeBuild to create the container image for FastQC in order to complete all work using only the AWS Management Console.

On the other hand, if Docker is available on your local PC, you can also create the container image in a local environment and push it directly to ECR. With this method, there is no need to create a CodeBuild project or an IAM role for CodeBuild.

Note that a container image created locally cannot be sent directly to AWS Batch. It needs to be saved to a container registry such as ECR so that AWS Batch can retrieve it.

Prerequisites
Prepare the following tools on your local PC.

• Docker
• AWS CLI
• AWS credentials for use with AWS CLI

Place the Dockerfile and Execution Script
Create a working directory and place the following 2 files inside it.

fastqc-batch/
├── Dockerfile
└── run_fastqc.sh

In CodeBuild, these files were encoded in Base64 to input their contents as a single line in the Build commands field. In a local environment, files can be created directly, so there is no need to use Base64.

In the directory where the files are placed, use Docker to build the container image.

Push the Container Image to ECR
Open the target repository in the ECR console and select "View push commands" at the top right of the screen to see commands corresponding to the account ID, region, and repository name you are using.

Push commands to ECR

The displayed commands include logging in to ECR, building the container image, tagging, and pushing to ECR. Basically, you can proceed with the push by executing them in the displayed order.

By default, latest is used as the image tag. To match the settings in this article, change latest to 0.12.1 in the tagging command and the push command.

Once the push is complete, open the ECR repository in the AWS Management Console. If the image with the 0.12.1 tag is displayed, it was successful.

For the AWS Batch Job definition, specify the image URI in the following format, the same as when created with CodeBuild.

AWSAccountID.dkr.ecr.us-east-1.amazonaws.com/rna-seq/fastq-test:0.12.1

ECR Push Permissions
The IAM user or IAM role executing the commands needs permissions to push images to ECR.

Mainly, permissions for the following operations are used. (Reference)

ecr:GetAuthorizationToken
ecr:BatchCheckLayerAvailability
ecr:InitiateLayerUpload
ecr:UploadLayerPart
ecr:CompleteLayerUpload
ecr:PutImage
ecr:BatchGetImage

If permissions are insufficient, errors such as AccessDenied will be displayed when running docker push.

Differences from the CodeBuild Method

With either method, the FastQC container image is ultimately saved to ECR. Therefore, the AWS Batch settings and execution method after the ECR push is complete do not change.

If Docker is already installed on your local PC, building locally and pushing to ECR has fewer steps. On the other hand, if you do not want to change the local environment or want to complete everything using only browser operations, using CodeBuild is more suitable.

4. Add ECR Push Permissions to the CodeBuild Role

Add an IAM policy to the CodeBuild service role so that CodeBuild can push images to ECR.
In the search field on the IAM role dashboard, type "codebuild" to find the role that was created when "Create a new service role" was selected earlier, and click the role name.

IAM policy settings screen

Click "Add permissions" → "Create inline policy" at the top right of the permissions policies section to open the policy editor.
Select the "JSON" tab at the top right of the policy editor and describe the policy content.

The following policy was set this time.

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Sid": "GetEcrAuthorizationToken",
     "Effect": "Allow",
     "Action": [
       "ecr:GetAuthorizationToken"
     ],
     "Resource": "*"
   },
   {
     "Sid": "PushFastqcImage",
     "Effect": "Allow",
     "Action": [
       "ecr:BatchCheckLayerAvailability",
       "ecr:GetDownloadUrlForLayer",
       "ecr:BatchGetImage",
       "ecr:InitiateLayerUpload",
       "ecr:UploadLayerPart",
       "ecr:CompleteLayerUpload",
       "ecr:PutImage"
     ],
     "Resource": "arn:aws:ecr:us-east-1:AWSAccountID:repository/rna-seq/fastq-test"
   }
 ]
}

This IAM policy directly specifies the ARN of the ECR repository in Resource. Therefore, the region name, AWS account ID, and repository name must match the repository created earlier.

5. Create IAM Roles for AWS Batch

In AWS Batch Fargate jobs, 2 types of IAM roles are used.

The names are similar, but the purposes are different.

The Execution role is used by Fargate to retrieve container images from ECR and send execution logs to CloudWatch Logs. On the other hand, the Job role is used when the running container accesses AWS services such as S3.

In this configuration, the Job role is used when the FastQC container retrieves FASTQ from S3 and saves analysis results to S3.

Execution Role

Attach the following AWS managed policy to the Execution role.

AmazonECSTaskExecutionRolePolicy

For the role name, use something like the following.

FastqcBatchExecutionRole

Now let's actually create the role.
First, click the "Create role" button at the top right of the IAM role dashboard to enter the role creation screen.

For the trusted entity type, specify "AWS service," and for the use case, specify "ECS Task."

For the permissions policy, search for and specify the AWS managed policy "AmazonECSTaskExecutionRolePolicy."

Creating the Execution role

Job Role

The role name for the Job role is as follows.

FastqcBatchJobRole

For this role as well, specify "AWS service" as the trusted entity type and "ECS Task" as the use case.

S3 access permissions are set not with an AWS managed policy but with an inline policy as follows.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBucket",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::bucket-name-account-id-region-an"
    },
    {
      "Sid": "ReadFastqInput",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::bucket-name-account-id-region-an/input_data/*"
    },
    {
      "Sid": "WriteFastqcOutput",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::bucket-name-account-id-region-an/output_data/*"
    }
  ]
}

Object read/write permissions are limited to the prefixes used this time. Only read is allowed for input_data/, and only write is allowed for output_data/.

6. Create the AWS Batch Compute Environment

Next, create the AWS Batch Compute environment.

The settings are as follows.

Compute environment creation screen

Maximum vCPUs

The "Maximum vCPUs" of the Compute environment is not the number of vCPUs allocated to a single job, but the upper limit on the number of vCPUs that can be used simultaneously across the entire environment.

On the other hand, the number of vCPUs used by one job is specified in the Job definition. For example, in an environment with a maximum of 4 vCPUs, if each job uses 2 vCPUs, a maximum of 2 jobs can run simultaneously. The 3rd and subsequent jobs wait until a running job finishes and vCPUs become available.

This time, since we are verifying operation with 1 sample, a maximum of 4 vCPUs is sufficient.

Also, setting the maximum vCPUs to 256 does not mean 256 vCPUs are always running.
However, if a large number of jobs are unintentionally submitted, it may scale up to that limit.
Considering costs, it is safer to keep it small at the verification stage.

After creation, confirm that the Compute environment is in the following state.

State: ENABLED
Status: VALID

Details of the Compute environment after creation

Why Not Use EC2?

When thinking about running applications on AWS, the first service that comes to mind is probably EC2 (Elastic Compute Cloud). Indeed, if you just want to run FastQC once, it is simpler to run it directly on a local PC or on EC2 without using AWS Batch.

The purpose of using AWS Batch in this article is not to make a single FastQC run more efficient. The goal is to build a foundation that allows multiple analysis jobs to be submitted to a queue in advance when the number of samples increases, and jobs to be automatically executed according to available computing resources.

Therefore, the environment is being built with AWS Batch in mind from the start.

7. Create a Job Queue

Next, create a Job queue to accept jobs.
Select "Job queues" from the left menu of AWS Batch, and start creating by clicking the "Create" button at the top right.

The settings are as follows.

Job queue creation screen

After creation, confirm the following state.

State: ENABLED
Status: VALID

Details of the Job queue after creation

8. Create a FastQC Job Definition

In the job definition, set the container image to execute, CPU, memory, IAM role, environment variables, and other settings.
This time, a job definition for Fargate was created.

For the image, specify the image URI pushed to ECR with CodeBuild.

AWSAccountID.dkr.ecr.us-east-1.amazonaws.com/rna-seq/fastq-test:0.12.1

Container image settings of the job definition

Environment Variables

Input/output information to be passed to the container was set as environment variables.
For paired-end, the following 4 variables are used.

For example, the values would be as follows.

INPUT_R1=s3://bucket-name-account-id-region-name-an/input_data/sample1_R1.fastq.gz
INPUT_R2=s3://bucket-name-account-id-region-name-an/input_data/sample1_R2.fastq.gz
OUTPUT_S3=s3://bucket-name-account-id-region-name-an/output_data/
THREADS=2

For single-end, set INPUT_R2 as follows.

INPUT_R2=NONE

Environment variables of the job definition

Since this was a verification with only 1 sample, the S3 URI is set directly in the job definition.
When processing multiple samples, it would be more convenient to be able to override the values at job submission time.
Leave the scheduling priority blank.

9. Submit the FastQC Job

Now that the job definition and Job queue are ready, select "Submit new job" from the "Jobs" screen of AWS Batch.

The settings are as follows.

Job submission screen

10. Check Processing in CloudWatch Logs

From the job details screen, open the log stream to check the container execution log in CloudWatch Logs.

In this log, the following processing can mainly be confirmed.

• Download R1 from S3
• Download R2 from S3
• Run FastQC
• Upload results to S3

If a job fails, you can first look at CloudWatch Logs to confirm at which step the error occurred.

CloudWatch Logs during FastQC execution

Finally, the AWS Batch job status became SUCCEEDED.

Job SUCCEEDED screen

11. Check the FastQC Report Output to S3

Finally, check output_data/ in the S3 bucket.
For paired-end, an HTML and ZIP are created for each FASTQ.

output_data/
├── sample1_R1_fastqc.html
├── sample1_R1_fastqc.zip
├── sample1_R2_fastqc.html
└── sample1_R2_fastqc.zip

FastQC files output to S3

Download the HTML file and open it in a local browser to view the standard FastQC report.

FastQC report opened in a browser

Good work!

This completes the quality check of FASTQ files using AWS Batch from the AWS Management Console.
We were able to go through the entire process of analyzing FASTQ files on S3 with AWS Batch and returning the results to S3.
It may take some time to configure at first, but you will be able to do it more smoothly as you get used to it.

Also, as explained in the next section, many of the resources created this time do not incur charges just by being retained. Some resources such as compute environments, job queues, and IAM roles can be reused, so you can prepare the environment with fewer steps next time.

12. Cleanup

After verification, delete any unnecessary resources.

• CloudWatch Logs log group
• Input FASTQ
• FastQC results
• CodeBuild project
• ECR repository

No Fargate charges if no jobs are running

Fargate charges are based on vCPU, memory, and other resources while a job is actually running.
Therefore, even if the AWS Batch Compute environment remains, no Fargate computing charges will be incurred as long as no jobs are running.

In addition, the following resources basically do not incur charges just by being retained.

• AWS Batch
  • Compute environment
  • Job queue
  • Job definition
• IAM roles and policies
• Default VPC
• Subnets
• Security groups

On the other hand, the following services may incur charges when data is stored.

• S3
• ECR
• CloudWatch Logs

Be careful with NAT Gateway

This time, we chose not to create a NAT Gateway and instead assigned a public IP to the Fargate job.
If you have created a NAT Gateway, charges will be incurred based on the time the NAT Gateway exists, even without any traffic. If it is no longer needed after verification, it must be deleted.

Summary

This time, we used AWS Batch and Fargate to analyze FASTQ files on S3 with FastQC.
The steps we carried out are summarized as follows.

• Upload FASTQ files to S3
• Create an ECR repository
• Build a FastQC container with CodeBuild
• Push the container image to ECR
• Create an IAM role for AWS Batch
• Create a Fargate Compute environment
• Create a Job queue and Job definition
• Submit the FastQC job
• Check the HTML report output to S3

By running one sample first, we were able to go through the following relationships in AWS Batch.

• Job definition: What to run and with which resources
• Job queue: Where to hold jobs waiting to run
• Compute environment: On which infrastructure to actually run jobs

This time we used a container image with FastQC built in, but by changing the software or execution script incorporated into the container, the same mechanism can be applied to other bioinformatics analyses. For example, read trimming, mapping, and expression quantification can also be executed on AWS Batch by preparing container images that include the corresponding software.