Skip to content

Commit

Permalink
docs: add README and configuration for pypi
Browse files Browse the repository at this point in the history
  • Loading branch information
ArthurVerrez committed Oct 12, 2023
1 parent 3c4ade2 commit 4a88d3d
Show file tree
Hide file tree
Showing 3 changed files with 33 additions and 48 deletions.
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include requirements.txt
79 changes: 31 additions & 48 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,92 +2,75 @@

## Description

`proxygetter` is a Python utility for scraping and filtering proxies from [sslproxies.org](https://www.sslproxies.org/). It is designed to be lightweight, fast, and customizable.
`proxygetter` is a Python library that provides a fast and customizable way to scrape, filter, and manage proxies. It's powered by asyncio and aiohttp to validate proxies asynchronously. Originally designed to scrape from [sslproxies.org](https://www.sslproxies.org/), it now supports customizable sources and multiple filters.

## Installation

You can install `proxygetter` via pip:
Install the package via pip:

```bash
pip install proxygetter
```

## Usage
## Features

### Basic Usage

To get a list of all available proxies:
### Proxy Management
Manage your proxies with ease using the `ProxyManager` class.

```python
from proxygetter import get_proxies
proxies = get_proxies()
print(proxies)
from proxygetter import ProxyManager
manager = ProxyManager()
```

### Features

#### Filter by Country Code

To get proxies from a specific country:
#### Asynchronous Filtering
Filter proxies based on their validity asynchronously.

```python
proxies = get_proxies(country_code='US')
valid_proxies = manager.filter_with_validity(url="http://example.com")
```

#### Filter by Anonymity

To get anonymous or elite proxies (accepted values are `'anonymous'` and `'elite proxy'`):
### Proxy Information
Access details about each proxy through the `Proxy` class.

```python
proxies = get_proxies(anonymity='anonymous')
proxy = valid_proxies[0]
print(proxy.get_requests_format())
print(proxy.get_selenium_format())
```

#### Filter by HTTPS support
### Advanced Filters

To get proxies that support HTTPS:
Get proxies using advanced filters like country code, anonymity, https support, Google compatibility, and last checked time.

```python
proxies = get_proxies(https=True)
filtered_proxies = manager.get_proxies(country_code='US', anonymity='elite proxy', https=True, google=True, last_checked_max=600)
```

#### Filter by Google support
### Fetch a Random Proxy

To get proxies that were checked to work with Google:
You can fetch a random proxy based on specified filters.

```python
proxies = get_proxies(google=True)
random_proxy = manager.get_random_proxy(country_code='US', https=True)
```

#### Filter by Last Checked Time

To get proxies that were last checked within a certain number of seconds:

```python
proxies = get_proxies(last_checked_max=600) # Proxies checked within the last 10 minutes
```
### Environment Configuration

### Advanced Usage
Configure default user agent and timeout using environment variables.

You can combine multiple filters:

```python
proxies = get_proxies(country_code='US', anonymity='elite proxy', https=True, google=True, last_checked_max=600)
```bash
export PROXY_URL_CHECKER_USER_AGENT=your_user_agent
export PROXY_URL_CHECKER_TIMEOUT=your_timeout_value
```

## License

This project is under the MIT License - see the [LICENSE](LICENSE) file for details.

## Tests

You can run the tests using the `unittest` framework with the command:

```bash
python -m unittest tests/test_proxygetter.py
```
## Upcoming Features

## Coming changes
- Proxy blacklisting
- Additional proxy databases
- Enhanced documentation and examples

- Ability to only get 1 proxy instead of a list
- Ability to test the proxies with a given url
- More databases than just sslproxies.org
Feel free to contribute or suggest improvements.
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ def read_requirements():
long_description_content_type="text/markdown",
packages=find_packages(),
install_requires=read_requirements(),
include_package_data=True,
classifiers=[
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
Expand Down

0 comments on commit 4a88d3d

Please sign in to comment.