Skip to content

Latest commit

 

History

History
257 lines (179 loc) · 9.47 KB

README.md

File metadata and controls

257 lines (179 loc) · 9.47 KB

libxget-js

Non-interractive, chunk-based, web content retriever

NPM Version NPM Downloads

NPM

Installing

Via NPM:

npm install libxget

This installs a CLI binary accessible with the xget command.

# Check if the xget command has been installed and accessible on your path
$ xget -v
v0.2.1

Usage

CLI

The xget command, utilizes the library to retrieve web content by its chunks according to specification

# Normal
xget https://google.com/doodle.png

# Write to output file
xget https://google.com/doodle.png image.png

# Piping output
xget https://myserver.io/runtime.log --no-bar | less

# Stream response in real time (e.g Watching the movie)
xget https://service.com/movie.mp4 | vlc -

Use --help to see full usage documentation.

Programmatically

// Node CommonJS
const xget = require('libxget');
// Or ES6 Modules
import xget from 'libxget';
// Or Typescript
import * as xget from 'libxget';

Examples

xget(
  'https://github.com/microsoft/TypeScript/archive/master.zip',
  {chunks: 10, retries: 10}
).pipe(fs.createWriteStream('master.zip'));

Get the master branch of the Typescript repository. With 10 simultaneous downloads. Retrying each one to a max of 10.

API

xget(url[, options])

XGETOptions extends RequestOpts: Object

  • chunks: <number> Number of chunked-simultaneous downloads. Default: 5
  • retries: <number> Number of retries for each chunk. Default: 5
  • timeout: <number> Network response timeout (ms). Default: 20000
  • start: <number> Position to start feeding the stream from. Default: 0
  • size: <number> Number of bytes to stream off the response.
  • hash: <string> Hash algorithm to use to create a crypto.Hash instance computing the stream hash.
  • use: <object> Key-value pairs of middlewares with which to pipe the response object through. keys are strings, values are Transformer generating functions (Alternatively, use the xget.use() method).
  • with: <object> Key-value pairs of middlewares with which to pipe the dataslice object through. keys are strings, values are functions whose return values are accessible within the store. (Alternatively, use the xget.with() method).

xget.store: Map

A map whose keys and values are tags and return types of content processed within the withStack of the xget object.

xget(URL)
  .with('variable', () => 5)
  .once('set', store => {
    /*
      `store` is a map whose key and values directly match tags and return types within
       > a with call or the with object in the xget options
    */
    console.log(store.get('variable')) // 5
  })
  .pipe(FILE);

xget.ended: Boolean

A readonly property that tells whether or not the xget instance has ended.

xget.loaded: Boolean

A readonly property that tells whether or not the xget instance has been loaded.

xget.bytesRead: Number

A readonly property that tells how many bytes has been processed by the underlying streams.

Class: XGETStream extends stream.Readable

The core multi-chunk request instance.

new xget.XGETStream(url[, options])

xget.getHash([encoding])

  • encoding: <string> The character encoding to use. Default: 'hex'
  • Returns: <Buffer> | <string>

Calculates the digest of all data that has been processed by the library and its middleware transformers. This, creates a deep copy of the internal state of the current crypto.Hash object of which it calculates the digest.

This ensures you can get a hash of an instancce of the data even while still streaming from the URL response.

xget.getHashAlgorithm()

Returns the hash algorithm if any is in use.

xget.use(tag, handler)

Add a named handler to the use middleware stack whose return value would be used to transform the response stream in a series of pipes.

The handler method is called after the stream is requested from and we start pumping the underlying request instances for a data response stream.

The core expects the handler to return a stream.Duplex instance. (A readable, writable stream) to transform or passthrough the raw data streams along the way.

// Example, compressing the response content in real time
xget(URL)
  .use('compressor', () => zlib.createGzip())
  .pipe(fs.createWriteStream(FILE))

xget.with(tag, handler)

Add a named handler to the with middleware stack whose return value would be stored within the store after execution.

xget(URL)
  .with('bar', ({size}) => progressBar(size)) // Create a finite-sized progress bar
  .use('bar', (_, store) => store.get('bar').genStream()) // Create a stream handling object that updates the progressbar from the number of bytes flowing through itself
  .once('set', store => store.get('bar').print('Downloading...'))
  .pipe(FILE);

ChunkLoadInstance: Object

  • min: <number> The minimum extent for the chunk segment range.
  • max: <number> The maximum extent for the chunk segment range.
  • size: <number> The total size of the chunk segment.
  • stream: <ResilientStream> A resilient stream that wraps around a request instance.

WithMiddlewareFn: Function

  • loadData: <object>
    • url: <string> The URL specified.
    • size: <number> Finite number returned if server responds appropriately, else Infinity.
    • start: <number> Sticks to specification if server allows chunking via content-ranges else, resets to 0.
    • chunkable: <number> Whether or not the URL feed can be chunked, supporting simultaneous connections.
    • chunkStack: <ChunkLoadInstance[]> The chunkstack array.

This handler is called immediately after metadata from URL is loaded that describes the response. That is, pre-streaming data from the HEAD like size (content-length), content-type, filename (content-disposition), whether or not it's chunkable (accept-ranges) and a couple of other criterias.

This information is passed into a handler whose return value is filed within the store referenced by the tag.

UseMiddlewareFn: Function

ClI Info

  • To avoid the terminal being cluttered while using pipes, direct other chained binaries' stdout and stderr to /dev/null
# Watching from a stream, hiding vlc's log information
xget https://myserver.com/movie.mp4 | vlc - > /dev/null 2>&1

Development

Building

Feel free to clone, use in adherance to the license and perhaps send pull requests

git clone https://github.com/miraclx/libxget-js.git
cd libxget-js
npm install
# hack on code
npm run build

License

Apache 2.0 © Miraculous Owonubi (@miraclx) <omiraculous@gmail.com>