Dmr-Source provides unified methods to achieve stream of data sources, such as local / remote (http / ftp) ...
- Source
- FileSource
- HttpSource
- FtpSource
- MultiSource
- SftpSource
- HadoopSource
Get file from ftp server to local
const fs = new FtpSource({
host: "127.0.0.1",
path: "/source-ftp-test.dict",
port: 21,
});
const writer = new FileSource().createWritableStream({path: "/home/work/a.log"});
fs.createReadableStream().pipe(writer);
Copy big file by FileSource
const fs = new FileSource({encoding: "utf8"});
const reader = fs.createReadableStream({path: "/home/work/a.log"});
const writer =fs.createWritableStream((option) => {
option.path = "/home/work/b.log";
return option;
});
reader.pipe(writer);
Link readables to one readable stream
const stream1 = new HttpSource({host: "localhost", path: "/a.log"}).createReadableStream();
const stream2 = new FileSource({host: "localhost", path: "/b.log"}).createReadableStream();
const ms = new MultiSource().add(stream1).add(stream2).add(() => {
return new FileSource({host: "localhost", path: "/c.log"}).createReadableStream();
});
ms.createReadableStream().on("data", (chunk) => console.log(chunk));
Events
const stream = new HttpSource({host: "localhost", path: "/404.page", timeout: 6000})
.createReadableStream();
stream.on("error", (err) => {
console.log("should be 404 error", err);
});
stream.on("end", (err) => {
console.log("never arrive end");
});