from pyspider.libs.base_handler import *
import re
class Handler(BaseHandler):
'''
this is a sample handler
'''
crawl_config = {
"headers": {
"User-Agent": "BaiDuSpider",
}
}
@every(minutes=24 * 60, seconds=0)
def on_start(self):
self.crawl('http://www.appletuan.com/', callback=self.index_page)
@config(age=10 * 24 * 60 * 60)
def index_page(self, response):
for each in response.doc('.topic_title a[href^="http://"]').items():
self.crawl(each.attr.href, callback=self.detail_page)
def detail_page(self, response):
return {
"url": response.url,
"title": response.doc('.header h1').text(),
}
返回的标题是乱码,不过同样用于v2ex却没有,这个怎么解决?
1
binux 2014-11-27 14:57:40 +08:00 1
```
doc = pyquery.PyQuery(response.text) return { "url": response.url, "title": doc('.header h1').text(), } ``` 这就是 lxml 的蛋疼之处,给它 unicode 它有的时候它不认,给它 bytes 它又处理不好 间 https://github.com/binux/pyspider/pull/24 |
2
hillw4h OP @binux 照你的方法修改之后正常了。谢谢。
不过坦白说,没懂这样改为什么就可以了,哈哈。(issue也没看懂 - -!!) |