From 1699a88e00c0f1a6e31433905f4f7a873706bfa5 Mon Sep 17 00:00:00 2001 From: Egor Dmitriev Date: Fri, 31 Jan 2020 00:49:44 +0100 Subject: [PATCH 1/8] [FEATURE] Updated seosnap dependencies --- seosnap-cacheserver | 2 +- seosnap-cachewarmer | 2 +- seosnap-dashboard | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/seosnap-cacheserver b/seosnap-cacheserver index a29415a..5575350 160000 --- a/seosnap-cacheserver +++ b/seosnap-cacheserver @@ -1 +1 @@ -Subproject commit a29415acf96f3f492998b36c5095f7319a6e20d5 +Subproject commit 55753508c52db78146e186e26e4d4ff5553dbb51 diff --git a/seosnap-cachewarmer b/seosnap-cachewarmer index 82606b6..f1cdf6e 160000 --- a/seosnap-cachewarmer +++ b/seosnap-cachewarmer @@ -1 +1 @@ -Subproject commit 82606b6c92937bb0be0f9e6066a85de71585e55e +Subproject commit f1cdf6ecbffbccf825761a6e61fb6569bb34d640 diff --git a/seosnap-dashboard b/seosnap-dashboard index 07f0fde..7d34785 160000 --- a/seosnap-dashboard +++ b/seosnap-dashboard @@ -1 +1 @@ -Subproject commit 07f0fde3ad624924a17ed1c7ac41bb8edf4b1dc6 +Subproject commit 7d3478519da7389da2c45ee4d24c5ab9ed6f6a96 From 0b1156403ee2ed23a704fb6380250bdd98ae3a50 Mon Sep 17 00:00:00 2001 From: Egor Dmitriev Date: Sun, 2 Feb 2020 13:08:22 +0100 Subject: [PATCH 2/8] [TASK] Updated firewall example to unblock docker. [TASK] Added pull command to update script --- examples/firewall.sh | 7 ++++--- update.sh | 3 +-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/examples/firewall.sh b/examples/firewall.sh index ed2434f..125eb9e 100644 --- a/examples/firewall.sh +++ b/examples/firewall.sh @@ -3,7 +3,8 @@ sudo sh -c "echo '{ \"iptables\": false }' >> /etc/docker/daemon.json && systemctl restart docker" sudo ufw default deny incoming -sudo ufw allow from myip to any port 22 -sudo ufw allow from myip to any port 8080 -sudo ufw allow from myip to any port 5000 +sudo ufw allow from to any port 22 +sudo ufw allow from to any port 8080 +sudo ufw allow from to any port 5000 +sudo ufw allow out on docker0 from 172.17.0.0/16 sudo ufw enable \ No newline at end of file diff --git a/update.sh b/update.sh index 48a8a5f..359b7e9 100755 --- a/update.sh +++ b/update.sh @@ -1,4 +1,3 @@ #!/bin/bash - - +git pull origin master --recurse-submodules \ No newline at end of file From e256c6c1d6336d76d90ed282a0e672352a4d526a Mon Sep 17 00:00:00 2001 From: Egor Dmitriev Date: Sun, 2 Feb 2020 13:09:40 +0100 Subject: [PATCH 3/8] [TASK] Split admin and model modules [BUGFIX] Fixed use of root domain for link generation --- seosnap-dashboard | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/seosnap-dashboard b/seosnap-dashboard index 7d34785..6c1b8f9 160000 --- a/seosnap-dashboard +++ b/seosnap-dashboard @@ -1 +1 @@ -Subproject commit 7d3478519da7389da2c45ee4d24c5ab9ed6f6a96 +Subproject commit 6c1b8f9fb500912451ae09adcde0206b68a7be17 From f63acd59880fcc028941f4f1ab5dc09ca0046d62 Mon Sep 17 00:00:00 2001 From: Egor Dmitriev Date: Sun, 2 Feb 2020 23:40:44 +0100 Subject: [PATCH 4/8] [FEATURE] Added queue creation and the view / update api --- seosnap-dashboard | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/seosnap-dashboard b/seosnap-dashboard index 6c1b8f9..5859d22 160000 --- a/seosnap-dashboard +++ b/seosnap-dashboard @@ -1 +1 @@ -Subproject commit 6c1b8f9fb500912451ae09adcde0206b68a7be17 +Subproject commit 5859d221e42f3452ff42218adbda76ea3a2597cd From ee0a627443ba5a6b4bbfa1412d72e8cc0f30b998 Mon Sep 17 00:00:00 2001 From: Egor Dmitriev Date: Sun, 2 Feb 2020 23:41:12 +0100 Subject: [PATCH 5/8] [FEATURE] Added queue admin interface --- seosnap-dashboard | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/seosnap-dashboard b/seosnap-dashboard index 5859d22..724c024 160000 --- a/seosnap-dashboard +++ b/seosnap-dashboard @@ -1 +1 @@ -Subproject commit 5859d221e42f3452ff42218adbda76ea3a2597cd +Subproject commit 724c024078b40efaf837cbb57300c3e174a1fb4e From 03c4d3ba29afa26e0773641d539373e0a39d7625 Mon Sep 17 00:00:00 2001 From: Egor Dmitriev Date: Sun, 16 Feb 2020 00:47:17 +0100 Subject: [PATCH 6/8] [FEATURE] Added queue cleaning --- seosnap-cachewarmer | 2 +- seosnap-dashboard | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/seosnap-cachewarmer b/seosnap-cachewarmer index f1cdf6e..68aeafc 160000 --- a/seosnap-cachewarmer +++ b/seosnap-cachewarmer @@ -1 +1 @@ -Subproject commit f1cdf6ecbffbccf825761a6e61fb6569bb34d640 +Subproject commit 68aeafc8b2d5f77441a3e974d912434cce81e894 diff --git a/seosnap-dashboard b/seosnap-dashboard index 724c024..edb2723 160000 --- a/seosnap-dashboard +++ b/seosnap-dashboard @@ -1 +1 @@ -Subproject commit 724c024078b40efaf837cbb57300c3e174a1fb4e +Subproject commit edb272313d762999abad4fd0c27366a951b5ef94 From 360772969a2158cbf1edfd1167065dd413187067 Mon Sep 17 00:00:00 2001 From: Egor Dmitriev Date: Tue, 18 Feb 2020 23:43:07 +0100 Subject: [PATCH 7/8] [FEATURE] Added click cli. Added site loading. --- seosnap-cacheserver | 2 +- seosnap-cachewarmer | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/seosnap-cacheserver b/seosnap-cacheserver index 5575350..4b876cc 160000 --- a/seosnap-cacheserver +++ b/seosnap-cacheserver @@ -1 +1 @@ -Subproject commit 55753508c52db78146e186e26e4d4ff5553dbb51 +Subproject commit 4b876cc8dcf2472fd441c1c4661f333776fd1315 diff --git a/seosnap-cachewarmer b/seosnap-cachewarmer index 68aeafc..d1a889a 160000 --- a/seosnap-cachewarmer +++ b/seosnap-cachewarmer @@ -1 +1 @@ -Subproject commit 68aeafc8b2d5f77441a3e974d912434cce81e894 +Subproject commit d1a889acecf157dc60c7ded484e21c00cbb7e825 From a544a09553a3f7b4d964fc8b964381a5b01120c4 Mon Sep 17 00:00:00 2001 From: Egor Dmitriev Date: Tue, 25 Feb 2020 23:37:14 +0100 Subject: [PATCH 8/8] [FEATURE] Updated dependencies and the readme for release 0.2.0 --- README.md | 42 +++++++++++++++++++++++++++++++++++++++--- seosnap-cacheserver | 2 +- seosnap-cachewarmer | 2 +- seosnap-dashboard | 2 +- 4 files changed, 42 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1a952e0..06b418e 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Setup for the whole seosnap stack including dashboard, cache server and cache wa page caching PWA's. # Installation -* Pull the repo +* Pull the repo (*note: the pull is recursive*) ``` git clone --recursive git@github.com:experius/SeoSnap.git ``` @@ -27,5 +27,41 @@ Cache directory ./cache ## Run cache warmer Make sure you have created a website via dashboard http://127.0.0.1:8080/seosnap/website/add/ ``` -docker-compose run cachewarmer -``` \ No newline at end of file +docker-compose run cachewarmer cache +``` + +## Usage cache warmer [See](https://github.com/experius/SeoSnap-Cache-Warmer/blob/master/README.md) +### Commands +#### Cache +Handles caching of pages associated to given website +``` +Usage: crawl.py cache [OPTIONS] WEBSITE_IDS + +Options: + --follow_next BOOLEAN Follows rel-next links if enabled + --recache BOOLEAN Recached all pages instead of not yet cached ones + --use_queue BOOLEAN Cache urls from the queue instead of the sitemap + --load BOOLEAN Whether already loaded urls should be scraped instead + --help Show this message and exit. +``` + +#### Clean +Handles cleaning of the dashboard queue +``` +Usage: crawl.py clean [OPTIONS] WEBSITE_IDS + +Options: + --help Show this message and exit. +``` + +### Examples +``` +# Cache the sitemap of website 1 +docker-compose run cachewarmer cache 1 + +# Cache requests in queue for websites 1 and 2 + dc run cachewarmer cache 1,2 use_queue=true + +# Clean the queue for websites 1 and 2 +docker-compose run cachewarmer clean 1,2 +``` diff --git a/seosnap-cacheserver b/seosnap-cacheserver index 4b876cc..7557f0f 160000 --- a/seosnap-cacheserver +++ b/seosnap-cacheserver @@ -1 +1 @@ -Subproject commit 4b876cc8dcf2472fd441c1c4661f333776fd1315 +Subproject commit 7557f0f64ad4e153e574f5f248272b5a18eb80fd diff --git a/seosnap-cachewarmer b/seosnap-cachewarmer index d1a889a..f2d38da 160000 --- a/seosnap-cachewarmer +++ b/seosnap-cachewarmer @@ -1 +1 @@ -Subproject commit d1a889acecf157dc60c7ded484e21c00cbb7e825 +Subproject commit f2d38da035ea806182174e48cdce369bc4ab74fc diff --git a/seosnap-dashboard b/seosnap-dashboard index edb2723..390120c 160000 --- a/seosnap-dashboard +++ b/seosnap-dashboard @@ -1 +1 @@ -Subproject commit edb272313d762999abad4fd0c27366a951b5ef94 +Subproject commit 390120c9aa15c609f8b02b9712cb1f5787c583b6