Chapter Download Rate Limit

[TsumiHokiro]

A few sources have lately introduced a few ways of blocking web crawlers (eg. myboxnovel, novelfull). Although they have blocked users, I have noticed that it's not instant blocking most of the times.
Those sites will usually block you after you have downloaded X chapters in less than Y time. I don't know what time they have set it but, from my experience, I have been able to download circa 200+ chapters in 10 seconds before being blocked sometimes (going as far as 230~.) (Second edit to reflect real number)
Now, other crawlers, Comics crawlers in particular, implement limits when you're downloading resources. This not only helps the user, but it also helps the source as well. I believe this would be helpful in our case here as well.
Before we get even more cases of sources implementing ways of blocking lncrawler, I think it would be interesting to implement some sort of chapter rate limiter. I believe that a rate of 2~5 chapters a second would be an interesting experience. 2 being a tad slow but not really a problem for novels without many chapters, and 5 just enough to test waters?
P.S: From tests I have done before, if you do suspend your download mid crawling (ctrl+z) and resume later when you haven't reached the limit you can bypass the blocking, which is how I have realised it'd be possible to bypass this with a slower download speed. (edit for English)

[dipu-bd]

There is already a limit in place. After gettings lots of feedback from different users, the limit is set to 10 per source.

https://github.com/dipu-bd/lightnovel-crawler/blob/master/lncrawl/core/taskman.py

[TsumiHokiro]

I don't think that is working as intended. I have tried selecting different sources, as before stated, and have been consistently blocked. It's not a matter of the sources not working, they were delivering chapter. The problem is, if the crawler was supposed to be be going through foo.com then bar.com and etc for the chapters, its not.
I'm using lncrawler 3.0.1 on both termux and a Linux machine and ran on the same issue.
Also, if I try to access the site I'm trying to download the chapters from I'll be blocked by a wall stating I've been weebcrawling them. I just don't see the limit per domain working here.

[lukegotjellyfish]

I run lncrawl with MAX_REQUESTS_PER_DOMAIN in \lncrawl\core\taskman.py set to 3 for NovelPub (4 produces 503 HTTP errors by the 200th chapter while 25 results with almost every request failing).

For ease of use, It'd be nice to have an argument to set this limit to adjust for different sources.

[TsumiHokiro]

My point exactly. Though if there was a lower number set (than 25 that is), it would reduce make it better for both sides. We'd experience less sites having the need to block us, at the same time we'd also not notice it in case they did implement such blocks in case of other crawlers.
If you, however, were to implement a case-by-case solution, as people come and complain of sources blocking them, it would soon spiral in a chain of source-fixing...
Also, having it flexible, be it during download option, or as a argument, would allow for people to regulate to their own needs.

[dipu-bd]

People who are using proxy usually do not have any problem with high limit.

I agree that this rate should be easily configurable by all. I'll add a new argument option for that.

[thepaulokill]

Can you teach me how to do this?