Crazy Ideas: OpenDAL Cache

[Xuanwo]

In the past, we attempted to implement a cache layer in OpenDAL, but it did not function effectively. Therefore, it is now necessary for us to revisit this area and identify potential improvements.

For example, can we port https://github.com/facebook/CacheLib to pure rust with opendal?

[Xuanwo]

Maybe we can try cache for parquet/orc/json directly instead a general cache lib.

[Ted-Jiang]

Ask a question, not read the code overall, what the unit of the cache of parquet (whole or page) ? 🤔

[Ted-Jiang]

Sorry, another question : Is there any case combine opendal with datafusion parquetExec 🤔

[Xuanwo]

Is there any case combine opendal with datafusion parquet

Yes! We have a project to make opendal work with object_store: https://github.com/apache/incubator-opendal/tree/main/integrations/object_store. And we are working with upstream to implement some services via OpenDAL.

object_store_opendal is nearly ready for using 😆 (some edge case not passed yet)

[Ted-Jiang]

Oh thanks! Nice to here this !

[Xuanwo]

Please let me know if there are any issues with integrating OpenDAL.

[Ted-Jiang]

Sure , if have a chance i will have a try!

[MrCroxx]

Hi, @Xuanwo . Could I have a try with foyer? Is there a guide to do it?

[Xuanwo]

Perfect! I think we can add a FoyerLayer directly.

The only thing we need to do is implement a Layer in opendal. You can refer to LoggingLayer as an example.

[xiaguan]

Regarding this issue, I've had some recent thoughts. Leveraging the capabilities of opendal, we can easily access almost all storage media, which means that writing a multi-layer caching system will be quite straightforward. Users can assemble their caching systems like building blocks, for example (fifo memory 1GB, lru memory 10GB, fifo ssd 1GB, fifo s3 express one 10TB). The caching algorithms will actually become more complex (promote to the upper level, reinsert, move to the lower level, evict), but that's not the main point. The key focus is on how to abstract these layers?

[Xuanwo]

The key focus is on how to abstract these layers?

Agreed. The challenge here is how to design our cache layer to support or switch different detailed caching algorithms while avoiding unnecessary complexity.

[xyjixyjixyji]

I think mimic the design of CacheLib is a good idea. We could in high level cache files through naming with url.

If we only support parquet I think it is sort of weird since we are operating on file granularity all the time.