Various stability enhancements

[motin]

Fixes #19

Note: Includes the changes from #18 currently. If #18 is merged first, this PR will be rebased on the latest master

[englehardt]

Overall this looks good to me. I merged #18 so we will need to rebase this before merging. I'm going to run a test crawl shortly with the new logging (openwpm/OpenWPM#434). I'll r+ if I don't find any issues during that.

[motin]

I merged #18 so we will need to rebase this before merging

Thanks! This PR is now rebased on current master.

[englehardt]

I ran through a bunch of test runs today with this. Thanks for the improvements!

One question: now that we've moved redis off of a preemptible node, do we still have the option to run crawls with preemptible nodes? I suspect the remaining concern is data loss from visits cached in the S3Aggregator.

[motin]

One question: now that we've moved redis off of a preemptible node, do we still have the option to run crawls with preemptible nodes? I suspect the remaining concern is data loss from visits cached in the S3Aggregator.

You are right, we can't yet reliably run crawls with preemtible nodes. I filed a tracking issue here: #21