融合了scrapy的架构理念和twisted的任务调度思想，目前只是个精简版的，开源地址
https://github.com/aware-why/multithreaded_crawler/，已附带demo演示的例子。
有兴趣的可以参与进来，有更好想法的人可以直接发起pull request，热烈欢迎大家的贡献。
multithreaded_crawler
A condensed crawler framework of “multithreaded model”
dependency
At present, the framework depends on nothing except for modules in the python standard libraries.
Usage
cd
threaded_spiderpython
run.py --help
You
will see a demo output by python
run.py, it crawls the sina.com.cn using five threads and has the crawling
depth limited to be 2 by default (It‘s tested in python2.7).
In
threaded_spider directory, there are extra log files whose name like
“spider.*.log” respectively generated using python
run.py --thread=* command.
Community
QQ
Group: 4704309
Your contribute will be
welcome.