Skip to content

Users' Choice! Backup your 🗝 too.
Compare
Choose a tag to compare
@joehand joehand released this 04 Aug 18:34
· 112 commits to master since this release
v13.8.0
8b25a43
This commit was signed with the committer’s verified signature.
joehand Joe Hand
GPG key ID: 22324E3F975FBBE6
Learn about vigilant mode.

13.8.0

With this release, we are adding an exciting feature that really showcases how powerful Dat is, selective sync. Using the CLI you can now specify which files you want to download either with an option or using the .datdownload file. dat sync will download and keep updated on the selected files. This means you can put large datasets into Dat but have control over what files you download where.

All the features for this release:

  • Selective Sync (#834)
  • Key management (#828)
  • Commands run faster via lazy required modules (#821)

And thanks to @G-Ray for fixing the .gitignore that was wrong but I kept forgetting to fix!

Selective Sync

We heard some feedback from many users, including Sloan Digital Sky Survey, that selective sync would be nice for very large dat archives. This new feature adds:

  1. dat clone <key> --empty : create an empty archive.
  2. dat sync <dir> --select=<remote-file>,<remote-file>,<remote-file>: download only selected files
    OR
  3. dat sync <dir> --select-from-file=<local-file-list.txt> will parse the file and download all files listed (separated by newlines). Default is .datdownload

For example, check out this short video for a demo.

1. Clone an "empty Dat

To start selective sync, you download an empty copy be using the --empty option.

❯ dat clone datproject.org/jhand/cat-mapper ./cat-pictures --empty

Dat successfully created in empty mode. Download files using pull or sync.

This creates a .dat directory and downloads the metadata.

2. Sync the files you want

Once you have the dat folder created, there are two ways to choose which files to sync.

You can specify the files directly in sync:

~/Desktop
❯ cd cat-pictures 

~/Desktop/cat-pictures
❯ dat sync --select=cat-1.jpg,cat-2.jpg,dat.json
dat v13.7.0
Syncing: 2 files (7.6 MB)

(Note: we are working on calculating the download progress. You can tell it is done when the download speed goes to 0.)

You can also create a .datdownload file inside your dat with the filenames or directories:

# .datdownload file
cat-1.jpg
cat-2.jpg
dat.json

When you run dat sync (without any options), Dat will only download the files or directories in the .datdownload.

We are really excited about getting this feature released! We hope it'll help users managing large datasets like the Sloan Digital Sky Survey team.

Backup and Import your keys

Managing your secret keys is a vital feature for Dat. Without the secret key, you cannot update a dat you've created. However, it is important that the keys do not get accidentally shared so we know we need to get this right.

We've started implementing some basic key management. With this release, you can:

  • dat keys - prints out dat key and discovery key
  • dat keys export - prints out secret key (does not remove ownership from existing dat)
  • dat keys import - prompts user for secret key to make a dat writable.
    These are still somewhat advanced key features but will at least make it possible to backup or transfer ownership of a dat.
Assets 2
Loading