I tried to connect to Amazon S3 using IICS CDI Amazon S3 V2 Connector

Shiwani Lawte

2024.06.12

Introduction

I'm Shiwani from the Data Analytics Division. This time, I would like to try connecting to Amazon S3 using the IICS CDI Amazon S3 V2 Connector. In conclusion, I was able create a mapping to read from an Amazon S3 source using Amazon S3 V2 Connector.

Advance Preparation

It is assumed that, IICS Cloud, IICS Secure Agent, and VPC are already configured. If you haven't done yet, please refer to this article: Building an Informatica Cloud Data Integration environment on AWS

Amazon S3 Connecter Construction

Please refer to the official documentation provided by Informatica for detailed information about the Amazon S3 V2 Connector. Introduction to Amazon S3 V2 Connector

Create an S3 bucket

Go to the Amazon S3 management console.Create an S3 Bucket in the same region as the EC2 instance where the IICS Secure Agent is installed. You can leave the settings as default.

In the created bucket, create a folder to store the Target Data. I have created a folder with the name as TICKIT and stored the sample data init.

Create an Amazon IAM Policy

AWS IAM authentication is used to securely control access to Amazon S3 resources. The EC2 role must have access to the S3 folder. You need to setup the IAM policy for the EC2 instance of the IICS Secure Agent so that IICS CDI can access the S3 Bucket you just created. You can configure an IAM policy through the AWS console.

Perform the steps as follows:

Go to the Identity and Access Management (IAM) Concole. Click on the "Roles" from the Dashboard.
Select the IAM role granted to the EC2 instance of the IICS Secure Agent.
Click On Add Permission in Permissions section from the setting screen of the IAM role granted to the EC2 instance of the IICS Secure Agent. Click on "Create inline policy".
Select "S3" as the service. You can also restrict the "actions" in detail, but this time, select "All S3 actions".
For "Resource", select "Specific" and in the "bucket" section, select "Add ARN".
You can Confirm and Edit the policy statement on JSON Policy editer.
Enter any name in the policy confirmation and the policy creation is completed. Once created, you can confirm that the policy is attached to the IAM role.

S3 Connector Configuration

While logged in to IICS, click Administrator Service. Perform the steps as follows:

Click "New Connection" on the Administrator Service Connection screen.
Enter the following values for each property:
- Connection Name: Any connector name
- Type: Amazon S3 V2
- Runtime environment: Secure Agent
- Folder Path: Bucket name to connect to
- RegionName: Bucket Region
Check whether the connection test is successful.

Connect To Amazon S3

Create a new Mapping in IICS CDI. This time we will create a mapping to read from an Amazon S3 source.

Perform the steps as follows:

For "Source Connection", select the S3 Connector you created earlier.
For the "Object," browse and select the S3 bucket and the specific object you want to use as the source.
Specify any additional properties such as file format, delimiter (for CSV files).
You can check the "Data Preview".
Verify the Source Fields of the Source data.

By following these steps, you can successfully connect to an Amazon S3 in IICS CDI. This allows you to integrate and transform data stored in S3 with other data sources and targets within the IICS environment.