To install, first clone, then
virtualenv -p python3 .
This requires at least python3.6.1
Then, do
. bin/activate
pip install -r requirements.txt
To create the database:
sqlite3 albums.db
To create tables:
python3 models.py
To scrape first ten pages of high-scoring albums pages:
python3 scraper.py
To scrape some other page
python3 $URL
To scrape deeper/less deep on high-scoring albums
python3 100
If you want to crawl deeper from a single starting point, change the global variable RECURSION_DEPTH at the start of scraper.py
If you want to use more or fewer Python threads, change MAX_WORKERS at the start of scraper.py