How to restore from Amazon ES Manual Snapshot

yusuke

2018.08.05

この記事は公開されてから1年以上経過しています。情報が古い可能性がありますので、ご注意ください。

Amazon ES has both Automated Snapshots and Manual Snapshots, but an Automated Snapshot cannot be used to restore to a new Amazon ES domain.　Therefore, if we want to restore to a new Amazon ES domain, we will need to restore from a Manual Snapshot.

In this post, we will introduce how to take a Manual Snapshot and how to restore from the Manual Snapshot.

Step1: Create S3 bucket

We create an S3 bucket to store the Manual Snapshot. We create it using the name es-test-index-repo via the S3 console.

We write down the S3 bucket ARN as it will be used later.

arn:aws:s3:::es-test-index-repo

Step2: Set up IAM policy

We use the IAM console to create the policy es-test-index-repo-policy from Policies -> CREATE policy as shown below.

In the Resource section of the policy, we specify ARN arn:aws:s3:::es-test-index-repo to refer to the S3 bucket we created earlier.

{
   "Version":"2012-10-17",
   "Statement":[
       {
           "Action":[
               "s3:ListBucket"
           ],
           "Effect":"Allow",
           "Resource":[
               "arn:aws:s3:::es-test-index-repo"
           ]
       },
       {
           "Action":[
               "s3:GetObject",
               "s3:PutObject",
               "s3:DeleteObject"
           ],
           "Effect":"Allow",
           "Resource":[
               "arn:aws:s3:::es-test-index-repo/*"
           ]
       }
   ]
}

We then create the IAM Role es-test-index-repo-role from Roles -> Create role in the IAM console.

We attach the policy created earlier and set the trust relationship as follows.

Trust Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

We also create an attach the following policy to the users who will be creating the Manual Snapshot repository.

es-backup-policy

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "iam:PassRole",
        "Resource": "arn:aws:iam::aws_account_id:role/es-test-index-repo-role"
    }
}

Step3: Register Manual Snapshot repository

Before taking a Manual Snapshot for the first time, we must register the Manual Snapshot repository.

It is necessary to sign the AWS request here. curl does not support AWS request signature.

There is sample code for Python, which we will refer as reference.

register-repo.py

AWS_ACCESS_KEY_ID = ''
AWS_SECRET_ACCESS_KEY = ''
host = ''　# ES domain endpoint
region = 'us-west-2'
path = '' # Name of the snapshot repository
bucket　= '' # Name of the S3 bucket
role_arn = '' # The ANR of the IAM role created earlier

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID =''
AWS_SECRET_ACCESS_KEY =''
region = 'us-west-2'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo'
url = host + path

payload = {
  "type": "s3",
  "settings": {
    "bucket": "s3-bucket-name",
    "region": "us-west-2",
    "role_arn": "arn:aws:iam::aws_account_id:role/es-test-index-repo-role"
  }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.text)

Execute

$ python register-repo.py
{"acknowledged":true}

Step4: Take a Manual Snapshot

If the access policy specifies an IAM user or role, we must sign the snapshot request. Since there is sample code in the documentation, we will use it as reference.

If you need a signature request, please refer to the Python code below; if you do not need a signature request, please refer to the curl example below.

snapshot.py

path # Specify the snapshot name

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-2'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo/my-snapshot-1'
url = host + path

r = requests.put(url, auth=awsauth)

print(r.text)

Execute

$ python snapshot.py
{"accepted":true}

curl

$ curl -XPUT 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/my-snapshot-2'
{"accepted":true}

Step5: Check snapshot repository

check_repository.py

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-2'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/?pretty'
url = host + path

r = requests.get(url, auth=awsauth)

print(r.text)

execute

$ python repository.py
{
  "es-test-index-repo" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "es-test-index-repo",
      "region" : "us-west-2",
      "role_arn" : "arn:aws:iam::aws_account_id:role/es-test-index-repo"
    }
  },
  "cs-automated" : {
    "type" : "s3"
  }
}

curl

$ curl -XGET 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot?pretty'
{
  "es-test-index-repo" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "es-test-index-repo",
      "region" : "us-west-2",
      "role_arn" : "arn:aws:iam::aws_account_id:role/es-test-index-repo-role"
    }
  }
}

Step6: Confirm snapshot in repository

check_snapshot.py

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-2'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo/_all?pretty'
url = host + path

r = requests.get(url, auth=awsauth)

print(r.text)

Execute

$ python check_snapshot.py
{
  "snapshots" : [ {
    "snapshot" : "my-snapshot-1",
    "uuid" : "**********aixSzBQFI_2g",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [ "index1", "index2" ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T02:12:52.086Z",
    "start_time_in_millis" : 1527214372086,
    "end_time" : "2018-05-25T02:12:55.745Z",
    "end_time_in_millis" : 1527214375745,
    "duration_in_millis" : 3659,
    "failures" : [ ],
    "shards" : {
      "total" : 10,
      "failed" : 0,
      "successful" : 10
    }
  }, {
    "snapshot" : "my-snapshot-2",
    "uuid" : "**********Svf55AYG0QuQ",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [ "index1", "index2" ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T02:13:36.289Z",
    "start_time_in_millis" : 1527214416289,
    "end_time" : "2018-05-25T02:13:38.326Z",
    "end_time_in_millis" : 1527214418326,
    "duration_in_millis" : 2037,
    "failures" : [ ],
    "shards" : {
      "total" : 10,
      "failed" : 0,
      "successful" : 10
    }
  } ]
}

curl

curl -XGET 'elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/_all?pretty'
{
  "snapshots" : [ {
    "snapshot" : "my-snapshot-1",
    "uuid" : "**********aixSzBQFI_2g",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [ "index1", "index2" ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T02:12:52.086Z",
    "start_time_in_millis" : 1527214372086,
    "end_time" : "2018-05-25T02:12:55.745Z",
    "end_time_in_millis" : 1527214375745,
    "duration_in_millis" : 3659,
    "failures" : [ ],
    "shards" : {
      "total" : 10,
      "failed" : 0,
      "successful" : 10
    }
  }, {
    "snapshot" : "my-snapshot-2",
    "uuid" : "**********Svf55AYG0QuQ",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [ "index1", "index2" ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T02:13:36.289Z",
    "start_time_in_millis" : 1527214416289,
    "end_time" : "2018-05-25T02:13:38.326Z",
    "end_time_in_millis" : 1527214418326,
    "duration_in_millis" : 2037,
    "failures" : [ ],
    "shards" : {
      "total" : 10,
      "failed" : 0,
      "successful" : 10
    }
  } ]
}

Step7: Restore from snapshot

We will try two patterns here.

Restore to same ES domain
Restore to other ES domain

Step7-1 Restore to same ES domain

If there is an index with the same name, the restore will fail so we need to first delete it. ※ Amazon ES does not support Elasticsearch _close API.

delete_index.py

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-1'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

# DELETE INDEX

path = 'my-index'
url = host + path

r = requests.delete(url, auth=awsauth)

print(r.text)

$ python delete_index.py
{"acknowledged":true}

curl

Delete all indices

curl -XDELETE'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_all'

Delete specific index

$ curl -XDELETE 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/index-name'

restore-all.py

The path specifies the snapshot name.

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-1'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo/my-snapshot-1/_restore'
url = host + path

r = requests.post(url, auth=awsauth)

print(r.text)

$ python restore.py
{"accepted":true}

restore-one.py

Restore specific index. Here we are specifying a specific index with payload.

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-1'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo/my-snapshot-1/_restore'
url = host + path

payload = {"indices": "my-index"}

headers = {"Content-Type": "application/json"}

r = requests.post(url, auth=awsauth, json=payload, headers=headers)

print(r.text)

$ python restore-one.py
{"accepted":true}

curl

Restore all indices

curl -XPOST 'elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/my-snapshot-1/_restore'

{"accepted":true}

Restore specific index

$ curl -XPOST 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/_restore' -d '{"indices": "my-index"}' -H 'Content-Type: application/json'

Step7-2 Restore to other ES domain

When restoring to another domain, we need to register the same snapshot repository in the other domain.

Step3: Register a Manual Snapshot repository in the other domain.

※Change host in register-repo.py and execute it.

register-repo.py

$ python register-repo.py
{"acknowledged":true}

Then restore to another domain. Change host and execute it.

Restore done with restore.py

$ python restore-all.py
{"accepted":true}

curl

curl -XPOST 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/my-snapshot-1/_restore'

Conclusion

We illustrated how to restore from a Amazon ES Manual Snapshot. We hope this blog will be helpful when restoring to a different domain or restoring from a snapshot created during maintenance.