diff --git a/README.md b/README.md index 6b6764e..a481b55 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,10 @@ ![logo](https://github.com/experius/SeoSnap/raw/master/assets/logo.png) -#SeoSnap Beta - Setup for the whole seosnap stack including dashboard, cache server and cache warmer used for prerendering and full page caching PWA's. # Installation -* Pull the repo +* Pull the repo (*note: the pull is recursive*) ``` git clone --recursive git@github.com:experius/SeoSnap.git ``` @@ -15,12 +13,6 @@ git clone --recursive git@github.com:experius/SeoSnap.git ``` docker-compose up --build -d && docker-compose down ``` -Wait 5 seconds and then build again -``` -docker-compose up --build -``` - -Everything should run now # Usage * Dashboard: http://127.0.0.1:8080/ (default login: snaptron/Sn@ptron1337) @@ -35,7 +27,7 @@ Cache directory ./cache ## Run cache warmer Make sure you have created a website via dashboard http://127.0.0.1:8080/seosnap/website/add/ ``` -docker-compose run cachewarmer +docker-compose run cachewarmer cache ``` ## Nginx @@ -57,4 +49,40 @@ When the crawler is started it connects with the dashboard api. It uses scrapy t The cache server is a simple file caching server. If a file exist with the content of the page it serves the html from the file. If not, it renders the requested url with rendertron and saves the html output in a file. To refresh the cache the cache-warmer uses PUT requests instead of GET. This will force update from the cache file. # Build with -![diagram](https://github.com/experius/SeoSnap/raw/master/assets/software.png) \ No newline at end of file +![diagram](https://github.com/experius/SeoSnap/raw/master/assets/software.png) + +## Usage cache warmer [See](https://github.com/experius/SeoSnap-Cache-Warmer/blob/master/README.md) +### Commands +#### Cache +Handles caching of pages associated to given website +``` +Usage: crawl.py cache [OPTIONS] WEBSITE_IDS + +Options: + --follow_next BOOLEAN Follows rel-next links if enabled + --recache BOOLEAN Recached all pages instead of not yet cached ones + --use_queue BOOLEAN Cache urls from the queue instead of the sitemap + --load BOOLEAN Whether already loaded urls should be scraped instead + --help Show this message and exit. +``` + +#### Clean +Handles cleaning of the dashboard queue +``` +Usage: crawl.py clean [OPTIONS] WEBSITE_IDS + +Options: + --help Show this message and exit. +``` + +### Examples +``` +# Cache the sitemap of website 1 +docker-compose run cachewarmer cache 1 + +# Cache requests in queue for websites 1 and 2 + dc run cachewarmer cache 1,2 use_queue=true + +# Clean the queue for websites 1 and 2 +docker-compose run cachewarmer clean 1,2 +``` diff --git a/examples/firewall.sh b/examples/firewall.sh index ed2434f..125eb9e 100644 --- a/examples/firewall.sh +++ b/examples/firewall.sh @@ -3,7 +3,8 @@ sudo sh -c "echo '{ \"iptables\": false }' >> /etc/docker/daemon.json && systemctl restart docker" sudo ufw default deny incoming -sudo ufw allow from myip to any port 22 -sudo ufw allow from myip to any port 8080 -sudo ufw allow from myip to any port 5000 +sudo ufw allow from to any port 22 +sudo ufw allow from to any port 8080 +sudo ufw allow from to any port 5000 +sudo ufw allow out on docker0 from 172.17.0.0/16 sudo ufw enable \ No newline at end of file diff --git a/seosnap-cacheserver b/seosnap-cacheserver index 5575350..7557f0f 160000 --- a/seosnap-cacheserver +++ b/seosnap-cacheserver @@ -1 +1 @@ -Subproject commit 55753508c52db78146e186e26e4d4ff5553dbb51 +Subproject commit 7557f0f64ad4e153e574f5f248272b5a18eb80fd diff --git a/seosnap-cachewarmer b/seosnap-cachewarmer index f1cdf6e..f2d38da 160000 --- a/seosnap-cachewarmer +++ b/seosnap-cachewarmer @@ -1 +1 @@ -Subproject commit f1cdf6ecbffbccf825761a6e61fb6569bb34d640 +Subproject commit f2d38da035ea806182174e48cdce369bc4ab74fc diff --git a/seosnap-dashboard b/seosnap-dashboard index 9e36704..390120c 160000 --- a/seosnap-dashboard +++ b/seosnap-dashboard @@ -1 +1 @@ -Subproject commit 9e36704474568be4551376220b36b6cd01b12dec +Subproject commit 390120c9aa15c609f8b02b9712cb1f5787c583b6 diff --git a/update.sh b/update.sh index 48a8a5f..359b7e9 100755 --- a/update.sh +++ b/update.sh @@ -1,4 +1,3 @@ #!/bin/bash - - +git pull origin master --recurse-submodules \ No newline at end of file