MongoDB: General backlog

[amotl]

About

A few backlog items picked up from previous patches.

• Support reading MongoDB Extended JSON and BSON files
• Support real-world data
• Optionally use a query instead of reading the whole collection
• Support reading JSON+BSON files from HTTP
• Cover more data types
• Performance improvements
• Bug: "created_at" : { "$date" : 1180690093000 } leads to:

    File "/path/to/commons_codec/transform/mongodb.py", line 124, in decode_extended_json
      is_date_numberlong = type_ == "$date" and "$numberLong" in value["$date"]
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  TypeError: argument of type 'int' is not iterable

• Bug: BulkProcessor does not properly account for INSERT OK, 0 rows affected (0.002 sec). That happens when inserting arrays including items deviating in structure {"data": [{"value": 4.42}, {"value": {"percentage": 55}}]}.

References

• MongoDB: Various fixes and improvements #216
• MongoDB: Improve type mapper, add support for more BSON types #236
• DMS/DynamoDB/MongoDB: Use SQL with parameters instead of inlining values #243
• MongoDB: More ad hoc fixes for supporting real-world data #255
• MongoDB: Improve support for reading JSON/BSON files #261