Skip to content

Commit

Permalink
Moved out deployment of redis-box from helper script
Browse files Browse the repository at this point in the history
  • Loading branch information
motin committed Aug 1, 2019
1 parent e4c8bac commit 4e3ac98
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 3 deletions.
9 changes: 7 additions & 2 deletions deployment/gcp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,12 +100,17 @@ Remember to change the `crawl.yaml` to point to `image: gcr.io/$PROJECT/openwpm`

## Deploy the redis server which we use for the work queue

This will launch a 1GB Basic tier Google Cloud Memorystore for Redis instance ($0.049/GB/hour):
Launch a 1GB Basic tier Google Cloud Memorystore for Redis instance ($0.049/GB/hour):
```
gcloud redis instances create crawlredis --size=1 --region=us-central1 --redis-version=redis_4_0
```

Next, use the following output:
Launch a temporary redis-box pod deployed to the cluster which we use to interact with the above Redis instance:
```
kubectl apply -f redis-box.yaml
```

Use the following output:
```
gcloud redis instances describe crawlredis --region=us-central1
```
Expand Down
1 change: 0 additions & 1 deletion deployment/load_site_list_into_redis.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ echo "DEL $REDIS_QUEUE_NAME:processing" >> joblist.txt
# awk #1 = Add the RPUSH command with the site value within single quotes
cat "$SITE_LIST_CSV" | sed '1!G;h;$!d' | sed "s/'/\\\'/g" | awk -F ',' 'FNR > 0 {print "RPUSH '$REDIS_QUEUE_NAME' '\''"$1","$2"'\''"}' >> joblist.txt

kubectl apply -f redis-box.yaml
kubectl cp joblist.txt redis-box:/tmp/joblist.txt
kubectl exec redis-box -- sh -c "cat /tmp/joblist.txt | redis-cli -h $REDIS_HOST --pipe"

Expand Down

0 comments on commit 4e3ac98

Please sign in to comment.