fastcdc: NodeJS bindings for @nlfielder's fastcdc-rs package.
This module implements the fast content defined chunking algorithm (FastCDC) described in the following paper:
W. Xia et al. "FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data Deduplication" Usenix 2016
Content defined chunking takes a blob of bytes (like a file for example) and cuts it into more or less uniform sized chunks along content boundaries. Because these chunk boundaries are determined by the contents of a file, not regular intervals they are robust to local modifications like insertions and deletions. This makes content defined chunking useful when deduplicating date, comparing files and synchronizing files over remote network storage.
const fastCDC = require('./index.js')
const buffer = new Uint8Array(10 * 1024)
for (let i = 0; i < buffer.length; ++i) {
buffer[i] = Math.random() * 256
}
console.log(fastCDC(buffer, {
min: 1024,
avg: 4096,
max: 65536
}))Output
[ 0, 2594, 6454, 8106, 9244, 10240 ]
If your system has an up-to-date build of nodejs and rust you should be able to install this package using
npm install fastcdc
Computes a list of content defined chunk boundaries using fastcdc-rs.
Arguments
bytesis either anArrayBufferor aUint8Arraycontaining the bytes to be hashedoptsis a set of optional arguments to the chunker. If it is a number, then it assumed to be the average chunk size in bytes (default is1024). Otherwise, if an object is passed then it is assumed to be a configuration with the following properties:minthe minimum chunk sizeavgthe average chunk sizemaxthe maximum chunk size
Returns An array of chunk boundaries (0 and bytes.length inclusive) representing the chunks of the file
(c) 2021 Mikola Lysenko. ISC License