搜了段 PY ( PY 交易)爬虫代码,想储备点壁纸好过冬。但是总会遇到 BLOCKED 的页面(应该是颜色图),也不跳过,就呆呆的站在那里等。我的天呐~这是地球爬虫该干的事吗。
是不是应该遇到请求超时 30 秒的就 pass 掉,换下一个了呢?
大家觉得呢?
丰富一下吧,我百度了发现网上并没有很好的采集 WALL 的壁纸爬虫,丰富之后,让更多的人收益吧~
虽然现在我觉得花瓣的图也挺好!
鞠躬~
当然, V2 现在风气不是辣么好,如果你不愿意回答问题本身的,请 pass 掉,别冷酸灵牙膏了。么么哒~
=====
import urllib2
from urllib2 import HTTPError, URLError
import math
import sys
import time
print "The start time: "+time.ctime()
x = input("Start value:")
y = input("Stop value:")
k = y-x
for i in range(x, y+1):
url = '
https://wallpapers.wallhaven.cc/wallpapers/full/wallhaven-'+str(i)+'.jpg'
req = urllib2.Request(url)
req.add_header("User-agent", "Mozilla/5.0 (Linux; Android 4.4.4; A31 Build/KTU84P; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.49 Mobile MQQBrowser/6.2 TBS/043015 Safari/537.36 V1_AND_SQ_6.6.9_482_YYB_D QQ/6.6.9.3060 NetType/4G WebP/0.3.0 Pixel/480")
name = 'D:\\pic\\'+str(i)+'.jpg'
try:
conn = urllib2.urlopen(req)
f = open(name, 'wb')
f.write(conn.read())
f.close()
if k < 100:
sys.stdout.write("Computing: [%s%s] %.2f%%\r" % ('#' * (i-x) , '-' * (y-i), (i-x)*100/k))
sys.stdout.flush()
time.sleep(0.01)
else:
sys.stdout.write("Computing: [%s%s] %.2f%% \r" % ('#' * ((i-x)/(k/50)) , '-' * ((y-i)/(k/50)), (i-x)*100.0/k))
sys.stdout.flush()
time.sleep(0.01)
except HTTPError ,e:
i = i+1
except URLError, e:
print "The server\'s something is wrong!"
break
print "\nThe finished time: "+time.ctime()
print 'pic saved'
……