你自己做网站或app，如果他们会用电脑的话，让他们当内容编辑。

我觉得先把代码写出来，把系统数据库等优化好，才去考虑性能问题。

学习，迷茫，迷失

技术不好可以培养，如果没有上进心，你是得好好考虑考虑。

没要下雨

正确。。。

额，原来是因为空格， 只需要用 s.strip()去掉空格就可以了。。。。

answers = question.all_answer()
question.save(answers, '/home/zhg/Pictures/result.txt', 'w')
但是这种似乎不太人性化啊
期望的效果是
answers.save(....)

@WKPlus 额，确实是加锁的关系，我理解较浅，锁一般用在什么地方？

import requests
from bs4 import BeautifulSoup
import multiprocessing

n = 0
url_list = ['http://ubuntuforums.org/forumdisplay.php?f=333', ]
for x in range(1, 50):
n += 1
raw_url = 'http://ubuntuforums.org/forumdisplay.php?f=333&page=%d' % n
url_list.append(raw_url)

def running(url, q, lock):
lock.acquire()
html = requests.get(url)
if html.status_code == 200:
html_text = html.text
soup = BeautifulSoup(html_text)
with open('/home/zhg/Pictures/cao.txt', 'a+') as f:
for link in soup.find_all('a', 'title'):
s = 'http://ubuntuforums.org/' + str(link.get('href')) + ' ' + str(link.get_text().encode('utf-8'))
f.writelines(s)
f.writelines('\n')
lock.release()

if __name__ == '__main__':
manager = multiprocessing.Manager()
p = multiprocessing.Pool(len(url_list))
q = manager.Queue()
lock = manager.Lock()
for x in url_list:
p.apply_async(running, args=(x, q, lock))
p.close()
p.join()
print "process ended"

@lhy360121 ubuntu

问题找到了，就是要爬取的网站有置顶的帖子，所以置顶的记录被重复了n次

我想试试能不能插入高亮 额。。。。。

import requests
from bs4 import BeautifulSoup
import threading
url_num = 0
url_list = ['http://ubuntuforums.org/forumdisplay.php?f=333',]
for x in range(1, 50):
url_num += 1
raw_url = 'http://ubuntuforums.org/forumdisplay.php?f=333&page=%d' % url_num
url_list.append(raw_url)
class MyThread(threading.Thread):
def __init__(self, func, args, name=""):
threading.Thread.__init__(self)
self.func = func
self.args = args
self.name = name
def run(self):
apply(self.func, self.args)
def running(url):
# lock.acquire()
html = requests.get(url)
if html.status_code == 200:
html_text = html.text
soup = BeautifulSoup(html_text)
with open('/home/zhg/Pictures/cao.txt', 'a+') as f:
for link in soup.find_all('a', 'title'):
s = 'http://ubuntuforums.org/' + str(link.get('href')) + ' ' + str(link.get_text().encode('utf-8'))
f.writelines(s)
f.writelines('\n')
# lock.release()
if __name__ == '__main__':
thread_list = [ MyThread(running, (url, ), running.__name__) for url in url_list ]
for t in thread_list:
t.setDaemon(True)
t.start()
for i in thread_list:
i.join()
print "process ended"
with open('/home/zhg/Pictures/cao.txt', 'r') as f:
f_list = f.readlines()
set_list = set(f_list)
for x in set_list:
if f_list.count(x) > 1:
print "the <%s> has found <%d>" % (x, f_list.count(x))

@clino 我的理解就是Queue模块是给多线程用的，多线程之间可以共享内存，而multiprocessing自带的Queue支持多进程间的通讯 ？

url里面这样写 Index.index() 写错了 a.index()

@clino 懂了，但是后面的分号有作用吗?我去掉分号也没有出现任何错误。

在学习threading。。。。。。