Comparing Engula and Alluxio

[HolyLow]

Hi @huachaohuang @tisonkun , I am interested in the Engula project and have a question on the project target. As I am new to the project, I might have some misunderstanding on it, so pls point it out wherever I am wrong.
From the provided design docs, I think Engula is an interesting project aiming at providing an elastic caching layer between the computation engines(e.g. ClickHouse, Presto, etc.) and underlying persistent storage(e.g. S3, HDFS, etc.), mainly by utilizing serverless architecture(k8s).
However, there is an existing popular apache project, Alluxio, for providing "memory speed virtual distributed storage system" which also serves as the caching layer just like Engula's target as I see it.

Therefore, I am curious if there are any thoughts on how to distinguishing Engula from Alluxio?
Currently I might have the following points, but maybe none of them is strong enough:

1. Engula is written in Rust, making it potentially less buggy.
2. Alluxio provides a FS protocol, which is general-purpose but might lose opt chances. Engula provides several more specific abstractions such as Journal, Storage, etc.

It would be great if you could kindly share your thoughts on the uniqueness of Engula compared with Alluxio.
Thanks a lot!

[tisonkun]

@HolyLow as a community evangelist more than a storage engine expert, I'd like to share these points:

1. Alluxio is not an Apache project. It's hosted by Alluxio Open Source Foundation, but its homepage is hosted under the copyright of Alluxio, Inc. I guess there are some trademark issues. Engula doesn't belong to a foundation. We're all volunteers and represent ourselves. It’s all Apache 2.0. There’s no Contributor License Agreement. Easy.

2. Engula is an extremely young project which means its immaturity but also possibility. I appreciate what Alluxio has implemented so far and we can learn from each other as open-source projects. There is no better or worse on maturity. If you prefer participating an extremely rapidly evolving as well as unstable project, to build its whole future with your intelligence, Engula would be a good fit. Alluxio, as a mature project, has a lot of existing project specific knowledge you need to know before working smoothly among multiple modules.

3. From the architecture graph I can say Alluxio uses a simple deployment model (master & worker) while Engula is designed to be more elastic and adaptive (different functional units). Also, as you point it out, Alluxio provides a FS protocol, while Engula is designed to have multiple engines to adapt different workloads.

And yes, Engula doesn't grow as we design yet, which is a more difficult goal. We'll appreciate any helps to make the goal come true.

[huachaohuang]

Thanks for your help. I actually do not intend to respond because this is comparing an apple to a pear. If I understand it right, Alluxio is a distributed cache, and Engula is a storage engine. That's different enough to me.

[HolyLow]

Thanks a lot for your reply. Sorry for my wrong expression of mixing up the apache license agreement and the apache project.

I am not a storage engine expert as well, as I have mainly focused on computation engines. Yet I still have great interest on Engula project (maybe for practicing Rust, for learning knowledge on storage engines). I'll try to find somewhere to start with.

[HolyLow]

If I understand it right, Alluxio is a distributed cache, and Engula is a storage engine. That's different enough to me.

I think engula is also a distributed cache engine after reading existing docs... Maybe because I misunderstood the design docs, or because the project is in an early stage.

Anyway, thanks for your reply.

[tisonkun]

Hi @HolyLow,

@huachaohuang's reply notice me that either Alluxio or JuiceFS is designed to be a data access layer between an existing storage system and the computation:

Note that Alluxio itself is not a persistent storage system.

But Engula is actually the storage engine, which means it owns its data and we have a file implementation for journal, storage, etc. See storage engine wiki for insights.

[HolyLow]

Hi @tisonkun @huachaohuang ,
thanks for your explanation. It seems I have some misunderstanding on the project, and now as a storage engine, the project seems much more challenging and interesting.