この記事は公開されてから1年以上経過しています。情報が古い可能性がありますので、ご注意ください。
DataPipeline
It is a service which that helps in sort, reformating, analyze, filtering, reporting data and deriving an outcome from it. It is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as On-Premises data sources, at specified intervals.
S3
Simple and popular AWS Service for storage. Replicates data by default across multiple facilities. It charges per usage. It is deeply integrated with AWS Services. Buckets are logical storage units. Objects are data added in the bucket. S3 has a storage class on object level which can save money by moving less frequently accessed objects to colder storage class.
DynamoDB
An Amazon AWS Web Services database which is fully managed, serverless, key-value NoSQL database, has high-performance applications at any scale. It has the following benefits such as built-in security, continuous backups, automated multi-region replication , in-memory caching, and data export tools
Demo
The following Roles must be created in the AWS account in advance inorder to make use of the AWS Data Pipeline
First IAM Role lets the pipeline take actions on behalf of the user
Log in to AWS and search IAM In search bar
Click on role
Click on create Role
In the AWS Service select DataPipeline
In the use cases click Data pipeline and click Permissions
Click next until you reach the Review page, write a role name and click create role
2nd IAM Role this provisions AWS resources on behalf of the user. For import and export of DynamoDB data the Amazon EMR cluster resource and Amazon EC2 instances are associated with that cluster resource.
Login to AWS and search IAM In search bar
Click on role
Click on create Role
In the AWS Service select Datapipeline
In the use case select EC2 Role for Data Pipeline
The attached Policies is already present
Click next until you reach Review page, write a role name and click create role
5
