Adapt to fsspec?

[martindurant]

I had not noticed this new addition to the file-system landscape - thank you for producing this?

Would you be interested in adapting the code to be compatible with fsspec? This would mean better Dask integration for the future, and provide your implementation with many features shared with the other file-systems, such as advanced walk/glob, random-access files, transactional writes... and of course mean less duplication of code.

[manish]

Sorry for the late response. Yes, that does indeed sound great to me. I will have a look this weekend and see how it can be adapted

[martindurant]

Given that datalake and blob now share an API, you may be able to do both in one :) I am happy to be involved in this, but I may disappear shortly on paternity leave.

[swt2c]

I agree, having fsspec compatibility would be great. I would also like it if the dask dependency could be optional, e.g., so I could use AzureBlobFileSystem standalone, for instance from pandas.

[manish]

I am currently on vacation and will be back on 9th. Will continue working on it.

[martindurant]

ndeed, neither s3fs nor gcsfs depend on dask

[asiron]

Hey everyone,
What's the status on this issue ? It would be indeed great if we could adapt to fsspec and add as an official dependency to Dask, have you done anything? maybe I could help ?

[martindurant]

Thanks for connecting @asiron . To you and everyone else on this thread:
The work in fsspec/adlfs#6 is essentially done, both datalake and blob are working, I believe! Please test and help write documentation for it.