[BUG] Fail to walk through a FlyteDirectory through os.listdir

[JiangJiaWei1103]

Describe the bug

Currently, FlyteDirectory doesn't support os.listdir with a remote S3 path (fortunately, a local path works!). The error is shown as follows:

Though FlyteDirectory provides listdir method itself, I just think os.listdir could provide a better UX.

Expected behavior

A FlyteDirectory with a remote S3 path can be walked through by os.listdir, as commented here

Additional context to reproduce

Create a demo dir on minio and add a text file first. Then, run the following script:

import os
from flytekit import task 
from flytekit.types.directory import FlyteDirectory 


S3_DIR = "s3://my-s3-bucket/demo


@task
def create_fd() -> FlyteDirectory:
    fd = FlyteDirectory(path=S3_DIR)

    entities = os.listdir(fd) 
    for e in entities:
        # Suppose there's no subdir
        with open(os.path.join(S3_DIR, e), "r") as f:
            print(f.read())

    return fd


if __name__ == "__main__":
    fd = create_fd()
    print(fd)

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[pingsutw]

People can use s3fs to list the files if they want: s3fs GitHub. I’m not sure if we need to support it, as it’s unclear whether we should return S3 paths or local paths when listing a Flyte directory

[JiangJiaWei1103]

That makes sense to me. Thanks for your quick reply!