Ingesting documents from third-party services

[homanp]

I've been playing around with the Unstructured Source Connects and the API I must say is a bit clunky.

Issues I've experienced

• The need to pass identifiers to files and folders renders the API more or less useless. GDrive, Dropbox, Box make it basically impossible for users to actually get the ID of the docs.

• The config of the different connects could be more seamless imo.

Suggestion

• Due to the above mentioned limitations of the source connectors I would opt for building these pipelines ourselves with the goal of only requiring the auth credentials from the user and a black list of file ids to be used for excluding documents.

TBD Topics

• Should we use the underlying strategy that Unstructured uses, namely downloading everything to a local folder and then processing or should we process on the fly?

• Are there any other frameworks/libs that could be utilised to facilitate ingestion? (Airbyte or similar).