New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

重复请求问题 #1

Open

flytosunmer opened this issue Jul 21, 2019 · 0 comments

flytosunmer commented Jul 21, 2019

当爬取深度设置为0时（或者其他级别），存在重复爬取一些公共资源url的问题，
问题原因疑似：将任务入队时，没有检测数据库中url是否已经存在？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment