【Report】AWS Greengrass Basic Workshop #reinvent #IOT204

This blog article is an English version of this.
Post by:Aimi Shioya, Lead Operations Engineer, Classmethod CANADA

Introduction

This report is about re:Invent 2017 workshop, IOT204 - AWS Greengrass Basic Workshop.

Outline

In the AWS Greegrass Basic Workshop, learn the concepts of AWS Greengrass. You will install, run, and configure the software. Then we show you how to connect a device to AWS IoT and to AWS Greengrass. In your account, learn to provision AWS Greengrass and create various communication scenarios. Route messages to locally connected devices, and to and from the cloud. Also learn how to use logging to get insights about what's happening on the AWS Greengrass Core. For this workshop, you need to have an AWS account created, and you need to bring your laptop.

Workshop Document

Workshop

This workshop will provide an understanding of the basics of AWS Greengrass.

■ What is learning?
・How to connect devices to AWS IoT and Greengrass
・Install Greengrass
・Device communication via Greengrass

The code for the Bootcamp was written for the Raspberry Pi or to be used on an EC2 instance. I used EC2 instance this time.

Prerequisites

Need the following tools/setup/knowledge:
・AWS Account
・SSH to login into Greengrass Core(EC2), SCP to copy files
・awscli
You don't need this if you launch an EC2 instances with CloufFormation as the awscli will be setup automatically.
・VPC with public subnet
・SSH key-pair ※If you use an EC2 instance you need this.

Check the region

Before you start with the next topics make sure that you work in an AWS region where Greengrass is available.
Go to the AWS console, choose AWS Greengrass.

Now Greengrass is available below:

• US East (N. Virginia)
• US West (Oregon)
• EU (Frankfurt)
• Asia Pacific (Sydney)
• Asia Pacific (Tokyo)

Oregon is the closest to my place, I choose Oregon.

AWS Greengrass Requirements

Minimum Hardware：
CPU single 1GHz
128MB RAM
x86 and Arm
Linux (Ubuntu, Amazon, Raspbian)

Create EC2 instance

Lanch EC2 instance with CloudFormation that AWS team prepared.

1. Select Template -> nothing to be changed(already select )
   
2. Next
   
3. Parameters
   3.1. Select SSHKeyName
   3.2. Select a SubnetID
   3.3. Select a VpcId
   
4. Next
   
5. Options -> nothing to be changed
   
6. Next
   
7. Review -> check "I acknowledge that AWS CloudFormation might create IAM resources." at the bottom of the page
   
8. Create
   
9. Hit the refresh symbol in the upper right section of the screen
   
10. Wait until the complete stack is created
    

Enable logging for AWS IoT

Go to the AWS IAM console and create Role.

1. Roles
   
2. Create role -> AWS service -> IoT -> Next: Permissions
   
3. Attached permissions policy:
   AWSIoTLogging, AWSIoTRuleActions
   
4. Next: Review
   
5. Set a role name
   
6. Create Role

After create role, go to the AWS IoT console.

1. Get started (only if no resources are provisioned)
   
2. Settings
   
3. CloudWatch Logs (if DISABLED) -> Edit
   
4. Change "Disable Logging" to "Debug (most verbose)"
   
5. Set role -> Select the role you just created
   
6. Update

The log files from AWS IoT are send to Amazon CloudWatch. The AWS console can be used to look at these logs.

Connect a thing to AWS IoT

・Connect a device to AWS IoT, a Greengrass Core to AWS IoT or a Greengrass Aware Device(GGAD)
・Using X.509 certificates and connectivity information.

A zip file will be provided containing keys and certificates and a script to install further required software.

1.Go to the AWS console

2.Onboard
3.Configuring a device -> Get started
4.Connect to AWS IoT -> Get started
5.Choose Linux/OSX and Python -> Next

6.Register a thing Name "ggad-1" -> Next step

7.Download a connection kit

8.Next step

SSH into the EC2 instances:

1. Create a directory named "ggad-1" (already done when CloudFormation was used)
   
2. Copy (scp) "connect_device_package.zip" into the directory "ggad-1"
   
3. Unzip the file "connect_device_package.zip

$ ls -la
total 28
drwxr-xr-x 2 ec2-user ec2-user  4096 Dec 10 01:15 .
drwx------ 6 ec2-user ec2-user  4096 Dec 10 00:49 ..
-rw-r--r-- 1 ec2-user ec2-user  3575 Dec 10 01:15 connect_device_package.zip
-rwxr-xr-x 1 ec2-user ec2-user 11591 Dec 10 00:49 GGBootcampPubSub.py
-rw-r--r-- 1 ec2-user ec2-user  2542 Dec 10 00:49 gg_discovery_api.py

$ unzip connect_device_package.zip
Archive:  connect_device_package.zip
  inflating: ggad-1.private.key
  inflating: ggad-1.public.key
  inflating: ggad-1.cert.pem
  inflating: start.sh

$ ls
connect_device_package.zip  ggad-1.private.key  start.sh
ggad-1.cert.pem             ggad-1.public.key

Launch start.sh as root:
※If the start.sh doesn't start and permission error occurs, change permission with chmod.

sudo ./start.sh

Send sensor data to AWS IoT

Send sensor data in JSON format to AWS IoT and watch the incoming data in the MQTT client.
For sending sensor data either from the SenseHat board or random function AWS team provides the following scripts:

・GGBootcampPubSub.py
・gg_discovery_api.py

If using Cloudformation, those scripts are already in the directory.

The script "start.sh" in the directory "ggad-1" must be modified to call GGBootcampPubSub.py instead of the example script from the AWS IoT Python SDK.

Before

python aws-iot-device-sdk-python/samples/basicPubSub/basicPubSub.py -e <YOUR_ENDPOINT>.<AWS_REGION>.amazonaws.com -r root-CA.crt -c ggad-1.cert.pem -k ggad-1.private.key

After

python GGBootcampPubSub.py -e <YOUR_ENDPOINT>.<AWS_REGION>.amazonaws.com -r root-CA.crt -c ggad-1.cert.pem -k ggad-1.private.key

Subscribe to the MQTT client built in to the AWS IoT console to see the messages which will be sent later on.

・sdk/test/Python
・$aws/events/#

Go to the AWS IoT Console.

1. Test
   
2. Subscription topic: sdk/test/Python -> Subscribe to topic
   
3. Subscribe to a topic
   
4. Subscription topic: $aws/events/# -> Subscribe to topic

Send sensor data to AWS IoT.
Launch start.sh via SSH console and watch at the MQTT client in the AWS IoT console.

Provision a second device ggad-2 the same way that you did for the device ggad-1. Need to connect as ggad-1 and ggad-2. As the scripts are taking command line arguments it is simple to make them send another clientId with the parameter "--clientId". This parameter must be added in "start.sh".

Change the clientId on a device.

ggad-1

python GGBootcampPubSub.py -e <YOUR_ENDPOINT>.<AWS_REGION>.amazonaws.com -r root-CA.crt -c ggad-1.cert.pem -k ggad-1.private.key --clientId ggad-1

ggad-2

python GGBootcampPubSub.py -e <YOUR_ENDPOINT>.<AWS_REGION>.amazonaws.com -r root-CA.crt -c ggad-2.cert.pem -k ggad-2.private.key --clientId ggad-2

To test if the clientId setting works go to the AWS IoT console an subscribe to
$aws/events/#

Provision a Greengrass Group

Create a Greengrass group and install a AWS Greengrass software on a device. A device connect to Greengrass Core and send data locally as well as to the cloud and receive messages from the cloud.

Create a Greengrass group. Go to the AWS Greengrass console.

1.Greengrass

2.Groups
3.Create Group
4.Use easy creation

5.Group Name: myFirstGGG

6.Next
7.Leave Name for Core untouched

8.Next
9.Create Group and Core

10.Download these resources as a tar.gz
11.If you are NOT using the EC2 instance created by CloudFormation download also the Greengrass software
12.Finish !!! Don't forget to click "Finish". Otherwise your group will not be created !!!

Verify in the AWS IoT console that your Greengrass Group has been created.
Greengrass -> Groups

Install Greengrass on your device

On the device which will act as your Greengrass Core the Greengrass software needs to be installed.

Create a Greengrass Service Role.
※AWS Greengrass requires access to AWS Lambda and AWS IoT data.

Go to the IAM console.

1. Roles
   
2. Create role
   
3. AWS service
   
4. Greengrass
   
5. Next: Permissions
   
6. Select AWSGreengrassResourceAccessRolePolicy
   
7. Next: Review
   
8. Role name: GreengrassRole
   
9. Create Role

Copy (scp) the tar.gz file that you have downloaded during the Greengrass group creation to your device into the directory /tmp.The tar.gz file contains keys, certificate and a configuration file (config.json) which will be used to configure your Greengrass Core.

Unpack the tar.gz in the directory /greengrass.

cd /greengrass
sudo tar zxvf /tmp/<unique-string>-setup.tar.gz

It's ready to start a Greengrass core.
Before you start the Greengrass daemon subscribe to the following topics. If the Core starts correctly you can observe activities on that topics.

Go to the AWS IoT console and choose Test.

① $aws/events/#
② $aws/things/#
③ #

Fire up Greengrass on EC2 instance.

$ cd /greengrass/ggc/core
$ sudo ./greengrassd start
Starting greengrass daemon
Greengrass successfully started with PID: 24809

Need to become root to access the log-directories on the Greengrass Core.

$ sudo su -
# cd /greengrass/ggc/var/log/system/
# tail -f runtime.log
[2017-12-10T05:26:43.673Z][INFO]-Started all system components
[2017-12-10T05:26:43.901Z][INFO]-Started Deployment Agent and listening for updates
[2017-12-10T05:26:43.901Z][INFO]-Started Deployment Agent and listening for updates
[2017-12-10T05:26:43.901Z][INFO]-Deployment agent connected to cloud
[2017-12-10T05:26:43.901Z][INFO]-Trying to subscribe to topic $aws/things/myFirstGGG_Core-gda/shadow/update/delta
[2017-12-10T05:26:43.917Z][INFO]-Subscribed to : $aws/things/myFirstGGG_Core-gda/shadow/update/delta
[2017-12-10T05:26:43.917Z][INFO]-Trying to subscribe to topic $aws/things/myFirstGGG_Core-gda/shadow/get/accepted
[2017-12-10T05:26:43.938Z][INFO]-Subscribed to : $aws/things/myFirstGGG_Core-gda/shadow/get/accepted

If there are any problems when starting AWS Greengrass check file "crash.log" for errors:

/greengrass/ggc/var/log/crash.log

Add devices to the Greengrass Group

The Greengrass Group consists currently only of a Core. In this step you will add devices to the Group. You will use the devices "ggad-1" and "ggad-2" which you used before. But these device will later be configured to connect to your Core instead of AWS IoT.

Go to the Greengrass console.

1.Groups
2.myFirstGGG

3.Devices
4.Add your first Device
5.Select an IoT Thing

6.Select ggad-1 -> Finish
7.Add ggad-2 in the same way to your Greengrass Group

Create a subscription

The two devices ggad-1 and ggad-2 which you just added to the Greengrass Group should communicate where ggad-1 acts as a publisher and ggad-2 as a subscriber.

To route messages between devices and therefore allow communication a so called subscription must be defined.

A subscription is a routing rule which consists of a source, a target an a topic filter. A subscription defines which source may communicate to which target on which topic.

Go to the Greengrass console.

1.Groups
2.myFirstGGG
3.Subscriptions

4.Add your first Subscription
5.Select a source -> Devices -> ggad-1
Select a target -> Devices -> ggad-2
Next

6.Optional topic filter -> sdk/test/Python
Next

7.Finish

Enable Logging for Greengrass

Enable Logging for Greengrass.
By default logging is not enabled for the Greengrass Core. Logging should be enabled to get insights what happens on the Core and also for troubleshooting purposes.

1. Groups
   
2. myFirstGGG
   
3. Settings
   
4. Scroll down
   Local logs configuration -> Edit
   
5. Add another log type
   
6. Check both "User Lambdas" and "Greengrass system"
   Update
   
7. What level of logs should be sent? -> Select Debug logs
   
8. Save

Logfiles on the Greengrass Core:

Log directory: /greengrass/ggc/var/log
System logs: /greengrass/ggc/var/log/system
Lambda logs: /greengrass/ggc/var/log/user/<AWS_REGION>/<ACCOUNT_ID>

Deploy the Greengrass Group to the device

After a Greengrass Group was created or changed the configuration needs to be deployed to the Greengrass Core.

On the device:

cd /greengrass/ggc/var/log/system
tail -f localwatch/localwatch.log *.log

In the AWS Greengrass Console

1.Groups
2.myFirstGGG
3.Actions -> Deploy

4.Choose Automatic detection

After some moments you should see activity in the logs on the device and the deployment status at the console.

Connect devices to the Greengrass Core

Now that the Greengrass core got a configuration set through a deployment devices will be connected to the Greengrass Core.

To connect devices to a Greengrass Core the connectivity information - ip address/DNS name and port - is required as well as the CA certificate which signed the Core's certificate.

These settings can be obtained automatically by devices through the discovery service in AWS IoT. To use the discovery service an IoT policy is required which allows the action greengrass:Discover.

To connect the devices ggad-1 and ggad-2 to the core by using the connectivity information from the discovery service the action greengrass:Discover must be added to the policy of each device.

Go to the AWS IoT console.

1.Manage
2.Things

3.Choose ggad-1
4.Security
5.Click the certificate

6.Policies
7.Click ggad-1-Policy
8.Edit policy document
9.in the Action section add "greengrass:Discover" -> Save as new version

New policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Subscribe",
        "iot:Connect",
        "iot:Receive",
        "greengrass:Discover" ★add
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

To change the policy document of ggad-2 same like ggad-1.

Take a look at the available Core endpoints:

Go to the AWS Greengrass Console.

1. Cores
   
2. myFirstGGG_Core
   
3. Connectivity

Connect devices to the Greengrass Core

The devices ggad-1 and ggad-2 will connect to the Greengrass Core and communicate locally. ggad-1 will act as publisher and ggad-2 as a subscriber which will write the data received from ggad-1 to STDOUT.

By default the devices are connecting to AWS IoT. To point them to a Greengrass Core the command line parameter "--connect-to greengrass" is used.

Modify for both devices the script "start.sh".

python GGBootcampPubSub.py -e <YOUR_ENDPOINT>.<AWS_REGION>.amazonaws.com -r root-CA.crt -c ggad-1.cert.pem -k ggad-1.private.key --clientId ggad-1 --connect-to greengrass

Open two terminals and start the devices with the script start.sh:

Terminal 1: start ggad-2
Terminal 2: start ggad-1
Terminal 1: you should see messages arriving from ggad-1

ggad-1

2017-12-10 06:53:47,965 - AWSIoTPythonSDK.core - INFO - publish: topic: sdk/test/Python
2017-12-10 06:53:47,965 - AWSIoTPythonSDK.core - INFO - arn:aws:iot:us-west-2:<AWS_account_ID>:thing/myFirstGGG_Core

ggad-2

2017-12-10 06:53:45,697 - AWSIoTPythonSDK.core - INFO - {"temperature": 34.21059259572323, "datetime": "2017-12-10T06:53:45", "pressure": 49.81736789234773, "device": "ggad-2", "humidity": 1145.6129113675813, "sensor": "Random"}
2017-12-10 06:53:45,697 - AWSIoTPythonSDK.core - INFO - publish: topic: sdk/test/Python
2017-12-10 06:53:45,697 - AWSIoTPythonSDK.core - INFO - arn:aws:iot:us-west-2:<AWS_account_ID>:thing/myFirstGGG_Core
2017-12-10 06:53:45,697 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync publish...

In the file "router.log" Greengrass logs the routing of messages.
/greengrass/ggc/var/log/system/router.log

router.log

[2017-12-10T06:53:40.691Z][INFO]-message received from (client/ggad-2) with subject (sdk/test/Python)
[2017-12-10T06:53:40.958Z][INFO]-message received from (client/ggad-1) with subject (sdk/test/Python)
[2017-12-10T06:53:40.958Z][INFO]-Message (client/ggad-1, sdk/test/Python) will be forwarded to client/ggad-2.
[2017-12-10T06:53:40.958Z][INFO]-Message (client/ggad-1, sdk/test/Python) forwarded to client/ggad-2, func arn:aws:lambda:::function:GGConnManager.

Device to cloud communication

In this exercise messages will be sent from a device (ggad-2) to the cloud. To route the messages accordingly from a device to the cloud another subscription has to be created.

Greengrass -> Group -> select subscription at myFirstGGG

Source: Device ggad-2
Target: IoT Cloud
Optional topic filter: sdk/test/Python
Deploy

Deploy Status:
pending -> progress -> succesfully

Go to the AWS IoT console and subscribe in AWS IoT to the topic sdk/test/Python.

On the device(ggad-2)

sudo ./start.sh

Look for incoming messages in the AWS IoT MQTT client.

Cloud to device communication

It is also possible to send messages from the cloud to a GGAD.In this exercise data from the cloud should be routed to the device ggad-1.

Greengrass -> Group -> select subscription at myFirstGGG

Source: IoT Cloud
Target: Device ggad-1
Optional topic filter: sdk/test/Python
Deploy

On the device(ggad-1)

sudo ./start.sh

Go to the AWS IoT console and subscribe in AWS IoT to the topic sdk/test/Python.

1. Publish to a topic
   
2. Publish: topic: sdk/test/Python
   You can leave the default message untouched
   
3. Publish to topic

On the device in the window/terminal where ggad-1 is running look for incoming messages.

2017-12-10 07:28:15,028 - AWSIoTPythonSDK.core - INFO - {"temperature": 23.628040064070234, "datetime": "2017-12-10T07:28:15", "pressure": 40.638825937419846, "device": "ggad-1", "humidity": 1117.8889468338257, "sensor": "Random"}
2017-12-10 07:28:15,028 - AWSIoTPythonSDK.core - INFO - publish: topic: sdk/test/Python
2017-12-10 07:28:15,028 - AWSIoTPythonSDK.core - INFO - arn:aws:iot:us-west-2:<AWS_account_ID>:thing/myFirstGGG_Core
2017-12-10 07:28:15,028 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync publish...

Done!

Monitor

Conclusion

I wanted to participate in the workshop which is related to IoT and I attended the Greengrass workshop.
(IoT related sessions and workshops were popular, I couldn't attend IoT Basic workshop because of the number participants were limited.)
Actually I couldn't finish the whole workshop. Later on I tried this boot camp on my own.
It took a long time to understand this workshop because it was my first time to experience AWS Greengrass and my information was limited about Iot workshop. Even though Greengrass is very interesting.
AWS team prepared Cloudformation for AWS Greengrass basic workshop, that's the reason it was an easy experience to go through with the AWS Greengrass compared to create EC2 instance from the beginning. You don't need to a lot of time to create EC2 instance and setup. I really reccomend to try this out for people who want deep understanding about AWS Greengrass.