How to change the number of error retries in AWS SDK for Python (Boto3)

2019.02.25

Although API calls may temporarily fail due to various factors, each AWS SDK implements automatic retries. In this blog, we will introduce how to change the number of error retries in the AWS SDK for Python (Boto3).

Please refer to the AWS Documentation about error retry and exponential backoff. - Error Retries and Exponential Backoff in AWS

Error Retries and Exponential Backoff in Boto3

Although retry processing is implemented automatically, the AWS SDK for Python (Boto3) seems to retry five times by default with exponential backoff.

Boto 3 Documentation | Config Reference

'max_attempts' -- An integer representing the maximum number of retry attempts that will be made on a single request. For example, setting this value to 2 will result in the request being retried at most two times after the initial request. Setting this value to 0 will result in no retries ever being attempted on the initial request. If not provided, the number of retries will default to whatever is modeled, which is typically four retries.

_retry.json

"retry": {
  "__default__": {
    "max_attempts": 5,
    "delay": {
      "type": "exponential",
      "base": "rand",
      "growth_factor": 2
    },

Change the error number of retries

We are able to change the error number of retry it by setting retries config (botocore.client.Config) when creating session or service clients.

Set retries to config when creating session

ex.) Set the error retries up to 10 times for API call

from boto3.session import Session
from botocore.client import Config


session = Session()
config = Config(
    retries=dict(
        max_attempts=10
    )
)client = session.client('ec2', config=config)

Set retries to config when creating client

ex.) Set the error retries up to 10 times for API call

import boto3
from botocore.config import Config


config = Config(
    retries=dict(
        max_attempts=10
    )
)
client = boto3.client('ec2', config=config)

Conclusion

Exponential backoff is an essential algorithm in retries. This time we introduced the concept with the AWS SDK for Python (Boto 3). However, we would recommend implementing retry logic incorporating exponential backoff even when you are not using the AWS SDK. In Python, there is a module called retrying.

References