Skip to content

Commit

Permalink
Add Proxy Pool service doc to readme
Browse files Browse the repository at this point in the history
  • Loading branch information
algo7 committed Dec 9, 2023
1 parent 9c2a750 commit 67802b6
Showing 1 changed file with 13 additions and 0 deletions.
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ A simple scraper for TripAdvisor reviews.
- [Visit the UI](#visit-the-ui)
- [Live Demo](#live-demo)
- [Proxy Pool](#proxy-pool)
- [Running the Proxy Pool](#running-the-proxy-pool)

## How to Install Docker:
1. [Windows](https://docs.docker.com/desktop/windows/install/)
Expand Down Expand Up @@ -87,3 +88,15 @@ The UI is accessible at `http://localhost:3000`.
A live demo of the container provisioner is available at [https://algo7.tools](https://algo7.tools).

# Proxy Pool
Proxy Pool is a docker image that runs both HTTP and SOCKS5 Proxies over OpenVPN (config to be provided by the user via docker bind mounts). `sockd`, `squid`, and `openvpn` client are managed by `supervisord` in the container. The service integrates with the Container Provisioner to provide a pool of proxies for the scraper to use. The container provisioner uses `docker-compose labels` to distinguish between different proxies. At this moment, the container provisioner only supports connecting to the Proxy Pool using HTTP proxies. Each service in the `docker-compose.yml` file represents a single proxy in the pool. The `docker-compose.yml` file for the proxy pool is located in the `proxy_pool` folder.

The Proxy Pool service can also be used directly with the scraper. Just make sure that the `PROXY_ADDRESS` environment variable is in the `docker-compose.yml` file for the scraper.

## Running the Proxy Pool
1. Pull the latest scraper Docker image
```bash
docker pull ghcr.io/algo7/tripadvisor-review-scraper/vpn_worker:latest
```
2. Create a docker-compose.yml file containing the configurations for each proxy (see the docker-compose.yml provided in the proxy_pool folder).
3. Place the OpenVPN config file of each proxy in the corresponding bind mount folder speicified in the docker-compose.yml file.
4. Run `docker-compose up` to start the container.

0 comments on commit 67802b6

Please sign in to comment.