Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

爬了3000多个subject 后面就不行了，是怎么回事啊，代码没看懂 #10

Closed
lxl315 opened this issue Mar 20, 2019 · 4 comments
Closed

爬了3000多个subject 后面就不行了，是怎么回事啊，代码没看懂 #10

lxl315 opened this issue Mar 20, 2019 · 4 comments
Labels
question

Comments

@lxl315
Copy link

lxl315 commented Mar 20, 2019

如题是因为页面上已经匹配到的url 都爬过的原因么？
@baabaaox
Copy link
Owner

@lxl315 如果不是 IP 被 ban，可以在这里添加一些初始的 url，原理就是无法枚举豆瓣 subject_id，可以通过它的推荐来获取尽可能多的 subject_id

@lxl315
Copy link
Author

lxl315 commented Mar 21, 2019

嗯，这个我看到了。试过。
另外我想改造下您的项目从一个ip接口，获取代理ip，然后爬取信息，如果ip被ban 了，就从接口里在获取一个ip，如果没有被ban，就一直用这个代理ip爬

@baabaaox
Copy link
Owner

@lxl315 既然你有 IP 接口那么随机取一个IP出来应该更好，长时间用一个 IP 很容易被让 IP ban掉，如果你维护一个可用的 IP 列表，随机取一个，放慢抓取的频率，这样才不容易被 ban

@lxl315
Copy link
Author

lxl315 commented Mar 22, 2019

好的。我去继续优化

@baabaaox baabaaox closed this as completed May 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@baabaaox @lxl315