This is a scraper written in Node.js and using Puppeteer that gets the videos served by Tum Conf services.
To install tum-conf-scraper, run:
$ npm install tum-conf-scraper
This module is written because videos hosted on Tum Conf are difficult to download and watchable only in the browser. By using the module video-scraper-core, I created this module, that allows those videos to be recorderd.
To scrape a video available at "https://tum-conf.zoom.us/rec/share/myvideo" and save it to "./saved.webm":
const { TumConfVideoScraper } = require('tum-conf-scraper');
async function main() {
// Create an instance of the scraper
const scraper = new TumConfVideoScraper('mypasscode', {
debug: true
});
// Launch the Chrome browser
await scraper.launch();
// Scrape and save the video
await scraper.scrape('https://tum-conf.zoom.us/rec/share/myvideo', './saved.webm');
// Close the browser
await scraper.close();
}
main();
To scrape and download more than one video:
const { TumConfVideoScraper } = require('tum-conf-scraper');
async function main() {
// Create an instance of the scraper
const scraper = new TumConfVideoScraper('mypasscode', {
debug: true
});
// Launch the Chrome browser
await scraper.launch();
// Scrape and save the first video
await scraper.scrape('https://tum-conf.zoom.us/rec/share/myvideo', './saved.webm');
// Scrape and save the second video
await scraper.scrape('https://tum-conf.zoom.us/rec/share/myvideo-bis', './saved_bis.webm');
// Close the browser
await scraper.close();
}
main();
To scrape and download in parallel more than one video:
const { TumConfVideoScraper } = require('tum-conf-scraper');
async function scrape(dest, link) {
// Create an instance of the scraper
const scraper = new TumConfVideoScraper('mypasscode', {
debug: true
});
// Launch the Chrome browser
await scraper.launch();
// Scrape and save the video
await scraper.scrape(link, dest);
// Close the browser
await scraper.close();
}
async function main() {
const tasks = [
['./saved.webm', 'https://tum-conf.zoom.us/rec/share/myvideo'],
['./saved_bis.webm', 'https://tum-conf.zoom.us/rec/share/myvideo-bis']
].map(([dest, link]) => scrape(dest, link));
await Promise.all(tasks);
}
main();
With custom options:
const { TumConfVideoScraper } = require('tum-conf-scraper');
async function main() {
// Browser options
const scraper = new TumConfVideoScraper('mypasscode', {
debug: true,
debugScope: 'This will be written as scope of the euberlog debug',
windowSize: {
width: 1000,
height: 800
},
browserExecutablePath: '/usr/bin/firefox'
});
await scraper.launch();
// Scraping options
await scraper.scrape('https://tum-conf.zoom.us/rec/share/myvideo', './saved.webm', { duration: 1000 });
await scraper.scrape('https://tum-conf.zoom.us/rec/share/myvideo-bis', './saved_bis.webm', {
audio: false,
delayAfterVideoStarted: 3000,
delayAfterVideoFinished: 2000
});
await scraper.close();
}
main();
...all the options can be seen in the API section or with the Typescript definitions.
The documentation site is: tum-conf-scraper documentation
The documentation for development site is: tum-conf-scraper dev documentation
The TumConfVideoScraper class, that scrapes a video from a "BBB WebKonferenze" and saves it to a file.
Syntax:
const scraper = new TumConfVideoScraper(passcode, options);
Parameters:
- passcode: A
string
that specifies the passcode to access the video page. - options: Optional. A
BrowserOptions
object that specifies the options for this instance.
Methods:
- setBrowserOptions(options: BrowserOptions): void: Changes the browser options with the ones given by the
options
parameter. - launch(): Promise: Launches the browser window.
- close(): Promise: Closes the browser window.
- scrape(url: string, destPath: string, options: ScrapingOptions): Promise: Scrapes the video in
url
and saves it todestPath
. Some ScrapingOptions can be passed.
The options given to the TumConfVideoScraper constructor, see video-scraper-core for more information.
The options passing to a scrape method, see video-scraper-core for more information.
There are also some error classes that can be thrown by this module, see video-scraper-core for more information.
The package is tested by using jest and ts-jest. The tests try for real to download some videos and check if they are saved, therefore, are not run in the CI because they are not headless.
- The default browser is Google Chrome on
/usr/bin/google-chrome
, because Chromium did not support the BBB videos. You can always change the browser executable path on the configurations. - By default (if the duration option is
null
), the duration of the recording will be automatically detected by looking at the vjs player of the page and by adding a stopping delay of 15 seconds. - This module can be uses only in headful mode.