Skip to content

Commit

Permalink
documentation updates for new url
Browse files Browse the repository at this point in the history
URL changed from istresearch.com to dmoztools.net for better stability
  • Loading branch information
Madison Bahmer committed Jan 19, 2018
1 parent 5064295 commit 5db560b
Show file tree
Hide file tree
Showing 4 changed files with 10 additions and 10 deletions.
2 changes: 1 addition & 1 deletion docs/topics/advanced/rediskeys.rst
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,6 @@ If you run the integration tests, there may be temporary Redis keys created that

- **cluster:test** - Used when testing the Kafka Monitor can act and set a key in Redis

- **test-spider:istresearch.com:queue** - Used when testing the crawler installation can interact with Redis and Kafka
- **test-spider:dmoztools.net:queue** - Used when testing the crawler installation can interact with Redis and Kafka

- **stats:crawler:<hostname>:test-spider:<window>** - Automatically created and destoryed during crawler testing by the stats collection mechanism settings.
6 changes: 3 additions & 3 deletions docs/topics/introduction/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -431,15 +431,15 @@ Which ever setup you chose, every process within should stay running for the rem

::

python kafka_monitor.py feed '{"url": "http://istresearch.com", "appid":"testapp", "crawlid":"abc123"}'
python kafka_monitor.py feed '{"url": "http://dmoztools.net", "appid":"testapp", "crawlid":"abc123"}'

You will see the following output on the command line for that successful request:

::

2015-12-22 15:45:37,457 [kafka-monitor] INFO: Feeding JSON into demo.incoming
{
"url": "http://istresearch.com",
"url": "http://dmoztools.net",
"crawlid": "abc123",
"appid": "testapp"
}
Expand All @@ -460,7 +460,7 @@ Crawl Request:

::

python kafka_monitor.py feed '{"url": "http://dmoz.org", "appid":"testapp", "crawlid":"abc1234", "maxdepth":1}'
python kafka_monitor.py feed '{"url": "http://dmoztools.net", "appid":"testapp", "crawlid":"abc1234", "maxdepth":1}'

Now send an ``info`` action request to see what is going on with the
crawl:
Expand Down
10 changes: 5 additions & 5 deletions docs/topics/kafka-monitor/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ JSON Object feeder into your desired Kafka Topic. This takes a valid JSON object

::

$ python kafka_monitor.py feed '{"url": "http://istresearch.com", "appid":"testapp", "crawlid":"ABC123"}'
$ python kafka_monitor.py feed '{"url": "http://dmoztools.net", "appid":"testapp", "crawlid":"ABC123"}'

The command line feed is very slow and should not be used in production. Instead, you should write your own continuously running application to feed Kafka the desired API requests that you require.

Expand Down Expand Up @@ -89,10 +89,10 @@ Feed an item

::

$ python kafka_monitor.py feed '{"url": "http://istresearch.com", "appid":"testapp", "crawlid":"ABC123"}'
$ python kafka_monitor.py feed '{"url": "http://dmoztools.net", "appid":"testapp", "crawlid":"ABC123"}'
2016-01-05 15:14:44,829 [kafka-monitor] INFO: Feeding JSON into demo.incoming
{
"url": "http://istresearch.com",
"url": "http://dmoztools.net",
"crawlid": "ABC123",
"appid": "testapp"
}
Expand All @@ -116,8 +116,8 @@ If you have a :ref:`Crawler <crawler>` running, you should see the html come thr
"response_headers": {
<headers omitted>
},
"response_url": "http://istresearch.com",
"url": "http://istresearch.com",
"response_url": "http://dmoztools.net",
"url": "http://dmoztools.net",
"status_code": 200,
"status_msg": "OK",
"appid": "testapp",
Expand Down
2 changes: 1 addition & 1 deletion docs/topics/rest/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ Feed a crawl request

::

$ curl scdev:5343/feed -H "Content-Type: application/json" -d '{"url":"istresearch.com", "appid":"madisonTest", "crawlid":"abc123"}'
$ curl scdev:5343/feed -H "Content-Type: application/json" -d '{"url":"http://dmoztools.net", "appid":"madisonTest", "crawlid":"abc123"}'

Feed a Stats request

Expand Down

0 comments on commit 5db560b

Please sign in to comment.