[FEATURE] Better alternative for "--rotate" in asynchronous access to mubeng

[rszyma]

Is your feature request related to a problem? Please describe.
Yes. I have 2 identical threads, both run an infinite loop that call a function that sends 2 GET requests. I need those pairs of GET requests to have the same proxy from mubeng used on them. Pseudocode to visualize:

class Thread:
    loop:
        // I need mubeng to use proxy 'x' from proxy list for 1st request
        request.get("http://example.com/resource1", proxy=localhost:mubeng_port)
        // I need mubeng to AGAIN use proxy 'x' from proxy list for 2nd request
        request.get("http://example.com/resource2", proxy=localhost:mubeng_port) 

thread1 = Thread()
thread2 = Thread()

Describe the solution you'd like
My proposition for implementation:

• pass to mubeng a number (let's call it "session id") in custom header in HTTP request
• mubeng finds that custom header in "session id", and picks a proxy in proxies array by index calculated by session_id % proxies_array_length

Describe alternatives you've considered

• --rotate <AFTER> option would not work in my case, because of asynchronous nature of threads
• running multiple instances of mubeng - would not work well because:
  • resource heaviness while scaling for hundreds of threads
  • I need one central proxy to be used by every thread

[dwisiswant0]

This is a great idea to improve. Will add new methods for this, like header, to use and/or control proxies based on offsets given in specific headers.

Meanwhile, we have released a new version to support synchronous requests for rotation guarantors since v0.7.0 and above.

[dwisiswant0]

It should already be on the dev branch. With header method, rotation is controlled by custom header X-Proxy-Offset: N. Can you please confirm, @nutel65?

Example workaround:

n = 0

class Thread:
    loop:
        request.get("http://example.com/resource1", headers={"X-Proxy-Offset": n}, proxy=localhost:mubeng_port)
        request.get("http://example.com/resource2", headers={"X-Proxy-Offset": n+1}, proxy=localhost:mubeng_port) 
        n = n + 2

thread1 = Thread()
thread2 = Thread()

[rszyma]

First of all, thank you for investing your time. Overall the result is mostly as I wanted it to be.
Tested with this Python code:

import requests

url = "http://ifconfig.me/ip"
proxy_url = "http://localhost:8000"
proxy = {"http": proxy_url, "https": proxy_url}

N = 0
r1 = requests.get(url, headers={"X-Proxy-Offset": str(N)}, proxies=proxy)
print(r1.text)
r2 = requests.get(url, headers={"X-Proxy-Offset": str(N)}, proxies=proxy)
print(r2.text)
r3 = requests.get(url, headers={"X-Proxy-Offset": str(N+1)}, proxies=proxy)
print(r3.text)

A few remarks:
For 0 < N < (mubeng_proxylist_size) it works predictably, but outside this range (or if a non-numeric string is passed as N) it falls back to using the first proxy from the list, which seems like an overlook (considering this is a proxy rotator).

I suggest using the following solution:

• Map every possible integer to a proxy. That could be done by N = N % mubeng_proxylist_size (I mentioned that in my previous post). That way you cannot make mistake using N out of range. Also this way you could map negative integers too.

Also, passing non-numeric string to the header should be treated exceptionally. My suggestion is to return an error in a similar fashion to "Proxy server error" returned on connection error.