How to download files from another machine in a cluster?

[ghost]

The example given shows how to download a file with a given URL via the supernode. How would I go about downloading a file which exists on the supernode or any other machine on the network?

[lowzj]

We cannot directly download files from machines in dragonfly cluster.
Files that exist in supernode are encoded by dragonfly, we cannot directly to use them.
And files on client machines that downloaded by dfget will be copied to $HOME/.small-dragonfly/data. These files' copies are using for transforming between dfget. They're temporary and will be deleted when no downloading request any more.

[ghost]

Ok so the use case is downloading files from the internet using P2P to speed it up. I for example, cannot use it for internal file sharing which I'm currently doing via rsync.

[zousheng]

@lowzj rsync sharing is still a common use case, is it possible or meaningful to add this feature into Dragonfly in the future?

[lowzj]

We should discuss this feature and find out which scenarios using rsync can be replaced by dragonfly.

[ghost]

To give you a use case:

I have a machine which downloads large data sets. Now this data has to be distributed to various nodes in the cluster. Not all of the data is distributed to all of the nodes.

For example I download huge data sets A, B, C & D.
Node 1 & 3 need A, B & D.
Node 2, & 4 need C & D.
In this case, I'm currently using rsync on each node to get data from the machine which downloaded all the data sets. I'd like to replace this with P2P sharing as there's an overlap of the data required by the different nodes. Obviously this is a very small example of the overall use case. I hope this helps clarify the use-case

[allencloud]

First of all, thanks sincerely for constantly using and supporting Dragonfly. We will try our best to keep Dragonfly better, and keep community and eco-system growing.

To get more feedback of Dragonfly's adoption in industry, would you mind helping comment in issue #219 to give more information about your enterprise's Dragonfly usage. Thanks a lot in advance. @asadsm

[ghost]

Sure, will do.