Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

namida: init at 0.5.0 #346285

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

namida: init at 0.5.0 #346285

wants to merge 2 commits into from
+122 −0

Conversation

jebriggsy
Copy link

namida

namida is a tool for fast file downloads over high-latency and/or unreliable networks. It uses UDP for bulk data transmission, together with a minimal TCP control stream to mediate retransmission of lost data.

namida is based upon Tsunami, a 2000s-era protocol and software suite for UDP-based file transmission. While Tsunami is still usable today, it has essentially not been updated since 2009, and has several problems that make it annoying to use nowadays. So, namida was created by first converting Tsunami's source code to Rust using C2Rust, manually converting the generated unsafe code to safe, more idiomatic Rust, and then making various improvements.

In the process some parts of Tsunami were removed. In particular, after 2006 Tsunami was primarily maintained by Finnish VLBI scientists (primarily Jan Wagner at Metsähovi Radio Observatory), who added support for VLBI-specific real-time networking hardware. The project does not have access to this hardware so the features were deprecated

Features

  • Dynamic UDP transfer rate adjustment to avoid overloading the client (it may still use all the available bandwidth, leaving none for other applications. It is highly recommended to use the --rate command to limit the transfer rate to a suitable value)
  • Optional lossy transfer mode: if some amount of data loss can be tolerated, namida can be configured to allow packets to be dropped, with an optional limit. By default, transfers are always lossless.

New features compared to Tsunami:

  • Simple CLI that allows everything to be done in one command invocation (in return, Tsunami's FTP-like interactive console has been removed)
  • Client-side NAT traversal: UDP packets can be received even if the client is behind NAT, without any additional manual configuration required.
  • Encrypted communication by default: snow is used to encrypt both TCP and UDP communication.
  • Resumption of interrupted transfers: if parts of a file to be downloaded are already present locally, those parts will be skipped by default

Usage

Run a namida server providing all files in the local directory:

$ namida serve

Run a namida server providing only some specific files:

$ namida serve file1.txt file2.txt

List the files a server has available:

$ namida dir --server example.com

Get a specific file from a server:

$ namida get --server example.com file1.txt

Get all files from a server:

$ namida get --server example.com --all

Many more options are available for the individual subcommands. Run namida help [command] to get more information.

Licensing information

namida is available under the same license as Tsunami (both the original Tsunami from Indiana University, and Jan Wagner's updated version), which is a permissive BSD-style license with the additional restriction that derivative programs may not be called “Tsunami” without permission from Indiana University. See LICENSE.txt on the project repository for the full license text.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@jebriggsy jebriggsy force-pushed the init-namida branch 2 times, most recently from 6a44df2 to 0e2f174 Compare October 4, 2024 03:27
romner-set
romner-set suggested changes Oct 13, 2024
View reviewed changes
pkgs/by-name/na/namida/package.nix Outdated
Comment on lines 12 to 20
# Package uses git history to include versioning in binary
src = fetchgit {
url = "https://github.com/meew0/namida.git";
rev = "782ab6baf679f830c4242bf071d8e85743fe77f7";
hash = "sha256-SVFlh603D7mFoN+N4M6v3TXRQWNOV5jMPefmz+qOMQU=";
leaveDotGit = true;
branchName = "namida";
};
nativeBuildInputs = [ git ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should likely be addressed upstream so we can use fetchFromGitHub instead. The package is also non-deterministic since its outputs differ based on time of compilation, which needs to be addressed before it can be added to nixpkgs.

Copy link
Author

@jebriggsy jebriggsy Oct 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have written a patch which checks whether the package is within a Nix build environment at build time. If so, it does not check git for a revision or generate a build time. Instead these values come from env in the package.nix file. I will be submitting a pull request upstream. This solves both the non-determinism and the need for leaveDotGit and fetchgit.

pkgs/by-name/na/namida/package.nix Outdated

rustPlatform.buildRustPackage rec {
pname = "namida";
version = "0.5";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

namida -V reports its version as namida 0.5.0, version should be the full version. Commit messages & PR title should also be changed.

Suggested change
version = "0.5";
version = "0.5.0";

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

pkgs/by-name/na/namida/package.nix Outdated
patches = [ ./client_get_mem_size_of.patch ];

meta = {
description = "Fast file transfer over high-latency connections via UDP";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
description = "Fast file transfer over high-latency connections via UDP";
description = "Client/server for fast file transfer over high-latency connections via UDP";

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

pkgs/by-name/na/namida/package.nix Outdated
own updated tools anyway.
'';
homepage = "https://github.com/meew0/namida";
changelog = "https://github.com/meew0/namida/commits/namida/";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit history is not a changelog.

Suggested change
changelog = "https://github.com/meew0/namida/commits/namida/";

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

pkgs/by-name/na/namida/package.nix Outdated
license = lib.licenses.free;
sourceProvenance = with lib.sourceTypes; [ fromSource ];
maintainers = with lib.maintainers; [ jebriggsy ];
platforms = with lib.platforms; [ "x86_64-linux" ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to work fine on aarch64 as well.

Suggested change
platforms = with lib.platforms; [ "x86_64-linux" ];
platforms = lib.platforms.linux;

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@jebriggsy jebriggsy changed the title namida: init at 0.5 namida: init at 0.5.0 Oct 14, 2024
@jebriggsy jebriggsy force-pushed the init-namida branch 3 times, most recently from 172831a to 8363ab4 Compare October 14, 2024 01:26
romner-set
romner-set suggested changes Oct 14, 2024
View reviewed changes
Copy link
Contributor

@romner-set romner-set left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nitpick about the set build date, otherwise that should be all. I've also noticed that the upstream is a pretty niche project (4 stars on github, not a lot of activity) so I'm not sure it's wanted in nixpkgs, but that's up to a committer to decide.

pkgs/by-name/na/namida/package.nix Outdated
# Set the git revision and build datetime declaritively
env = {
GIT_HASH = src.rev;
NAMIDA_COMPILE_DT = "2024-10-13 12:00:00"; # Date this package was committed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably better to make this a generic value like the unix epoch.

Suggested change
NAMIDA_COMPILE_DT = "2024-10-13 12:00:00"; # Date this package was committed
NAMIDA_COMPILE_DT = "1970-01-01 00:00:00";

Copy link
Author

@jebriggsy jebriggsy Oct 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timestamp fixed.

Niche

It is a bit niche, but I think it may be a bit more fair to look at the interest in the Tsunami protocol in general. It is used a bit more in internal data center networks and other situations where TCP can be congested or otherwise constrained (satellite based networks for example).

Neither use case is likely to generate much activity in public repositories.

@jebriggsy
namida: init at version 0.5.0
a88b7da 
namida is a tool for fast file downloads over high-latency and/or unreliable networks. It uses UDP for bulk data transmission, together with a minimal TCP control stream to mediate retransmission of lost data.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@romner-set romner-set romner-set approved these changes

Assignees
No one assigned
Labels
8.has: maintainer-list (update) 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 10.rebuild-linux: 1 11.by: package-maintainer 12.approvals: 1 12. first-time contribution backport release-24.05 Backport PR automatically needs_merger
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jebriggsy @romner-set @NixOSInfra