feat: add exponential backoff retry for ProvisionedThroughputExceeded / ThrottlingException / InternalServerError

[yoshidan]

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

add exponential backoff retry for ProvisionedThroughputExceeded / ThrottlingException / InternalServerError

Issue #, if available:
#195

Description of changes:

Added the configuration to retry batch_write_item_api.

Configuration

Since retry_batch_write_item is optional, it can be undefined.
In this case, no retry is performed.

dy config dump

---
tables: ~

---
using_region: ~
using_table: ~
using_port: ~
retry: ~

---
tables: ~

---
using_region: ~
using_table: ~
using_port: ~
retry:
  default:
    initial_backoff: ~
    max_backoff:
      secs: 10
      nanos: 0
    max_attempts: 20
  batch_write_item: ~

Execution Logs

Output retry logs with warn level.

ProvisionedThroughputExceeded

9675 items processed (7.86 items/sec)[2024-02-20T06:49:02Z WARN  dy::batch] Retry batch_write_item : The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
[2024-02-20T06:49:12Z WARN  dy::batch] Retry batch_write_item : The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
9950 items processed (154.91 items/sec)[2024-02-20T06:49:34Z WARN  dy::batch] Retry batch_write_item : The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
[2024-02-20T06:49:44Z WARN  dy::batch] Retry batch_write_item : The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
10000 items processed (7.89 items/sec)

ThrottlingException

375 items processed (13.29 items/sec)[2024-02-27T00:51:18Z WARN  dy::batch] Retry batch_write_item : Request ID: Some("O3R1LLIR8E56GL9UA5N37HEV53VV4KQNSO5AEMVJF66Q9ASUAAJG") Body: {"__type":"com.amazon.coral.availability#ThrottlingException","message":"Throughput exceeds the current capacity of your table or index. DynamoDB is automatically scaling your table or index so please try again shortly. If exceptions persist, check if you have a hot key: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html"}
700 items processed (13.95 items/sec)[2024-02-27T00:51:54Z WARN  dy::batch] Retry batch_write_item : Request ID: Some("8TCT72BODCGFIKBVILA25P369RVV4KQNSO5AEMVJF66Q9ASUAAJG") Body: {"__type":"com.amazon.coral.availability#ThrottlingException","message":"Throughput exceeds the current capacity of your table or index. DynamoDB is automatically scaling your table or index so please try again shortly. If exceptions persist, check if you have a hot key: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html"}
750 items processed (8.45 items/sec)[2024-02-27T00:52:08Z WARN  dy::batch] Retry batch_write_item : Request ID: Some("5E6T40L360OAF6P4MFEE2S0RINVV4KQNSO5AEMVJF66Q9ASUAAJG") Body: {"__type":"com.amazon.coral.availability#ThrottlingException","message":"Throughput exceeds the current capacity of your table or index. DynamoDB is automatically scaling your table or index so please try again shortly. If exceptions persist, check if you have a hot key: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html"}

[StoneDot]

I apologize for the delayed response.

First, thank you for bringing the excellent improvement to dynein. As you mentioned in the issue, exponential backoff enhances stability for batch writing. I am willing to merge this pull request. However, I would like to request some modifications because we need to maintain compatibility after the migration to AWS SDK for Rust, which is currently in progress.

I have left some comments requesting modifications. Could you please review these comments?

[yoshidan]

@StoneDot We have corrected the problem and would appreciate it if you could check again.

[StoneDot]

Thank you for addressing my comments. I appreciate your effort.

To merge this PR, we need a minor modification to pass the test and a change to maintain compatibility with Backon and AWS SDK. Also, please remember to squash these changes into a single commit after the review. It helps us track the change quickly.

The change of test code is required because we changed the structure of the configuration file. See the test code.

I believe it is pretty close to merging this.

[StoneDot]

I'm sorry for forgetting to mention this. You need to include the following message in the summary of the pull request.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[StoneDot]

@yoshidan Hello. I eagerly await your return so I can merge the pull request. I'm looking forward to moving forward with this. If you have any technical issues, please don't hesitate to let me know. I will assist you in completing the review process and merging this pull request.

--
もし英語でのコミュニケーションで細かい話をすることが難しい場合は、日本語でもかまわないのでご共有いただけると幸いです。もし記録として残しておくと有用そうな内容がある場合は、私の方で英訳を残しておくこともできるかと思います。

[yoshidan]

@StoneDot Sorry for the late response. Now I fixed the code. Please review again.

[StoneDot]

Thank you so much for your quick response and fix. I know everyone has many things to do and is busy, so please do not mind it.

All things seem excellent to me. Thank you for contributing it. I will approve this and merge it.