DynamoDB: General backlog

[amotl]

About

Coming from GH-226, GH-247, GH-252, GH-263, and others, there are a few backlog items to be resolved.

Items

General

• DynamoDB: Properly loading data from tables using single-table design #232
• Transformations.
  That's another item transformation idea picked up from an example program.

  if 'id' in item and not isinstance(item['id'], str):
      item['id'] = str(item['id'])

  Please advise if this is sensible in all situations, or if it's just a special case.

Full Load

• Processing records in batches.
  DynamoDB: Add pagination support for full-load table loader #252
• Scan by query instead of doing a full scan.
• Improve error handling by using BulkProcessor, like MongoDB I/O is doing it.
  MongoDB: Improve error handling wrt. bulk operations vs. usability #262
• Using DynamoDB Scan+Pagination needs a software test.
• Decouple ingress pagination from egress batching?
• Implement Parallel scan.

Change Stream

• Improve error handling of processor Lambda by storing invalid records into a dead letter box.
• Consume from AWS Kinesis Stream ARN instead of stream name only, to make it work across organization boundaries, for example through connectivity using AWS PrivateLink. Via DynamoDB: Add ctk load table interface for processing CDC events #247.

Manual Testing

Other than having solid software tests, which are included into corresponding PRs, the DynamoDB I/O subsystem must be manually tested for semantic soundness and good usability, specifically but not exclusively related to full-load -> cdc convergence.

In the best possible scenario, this would happen on real-world data instead of synthesized or other data coming from testing datasets, because those are mostly uniform and homogeneous. In this spirit, the former is crucial for discovering edge case situations, and hopefully improving them.