开发环境建议：ubuntu+eclipse （windows + cygwin + eclipse不推荐） 第一步：下载
http://archive.apache.org/dist/nutch/
从上述站点下载src和bin两个压缩文件
wget 'http://archive.apache.org/dist/nutch/1.7/apache-nutch-1.7-bin.tar.gz'
wget 'http://archive.apache.org/dist/nutch/1.7/apache-nutch-1.7-src.tar.gz' 第二步：解压
tar zxvf apache-nutch-1.7-bin.tar.gz
解压出一个 apache-nutch-1.7 文件夹
重命名： mv apache-nutch-1.7 apache-nutch-1.7-bin tar zxvf apache-nutch-1.7-src.tar.gz
解压出一个 apache-nutch-1.7 文件夹
重命名： mv apache-nutch-1.7 apache-nutch-1.7-src 第三步：组合
将apache-nutch-1.7-bin/lib中的所有jar包拷贝到apache-nutch-1.7-src/lib中
cp apache-nutch-1.7-bin/lib/* apache-nutch-1.7-src/lib/
将apache-nutch-1.7-bin/conf中的配置文件覆盖apache-nutch-1.7-src/conf中 第四步：导入eclipse
eclipse : File -- New -- Java Project
这一步完成了将源码（而非工程）导入eclipse
注解：笔者以前用的eclipse版本有import project from source ，但这个版本没有，只有import project from existing project.而我们只有src文件 点击NEXT
找到 conf 文件夹 ，然后点击 Add Folder 'conf' to build path
defautl output 设置为 apache-nutch-1.7/bin 点击Finish 第四步：一些小BUG
此时会发现工程有错误（红色的小叉叉），这是因为缺少引用导致的。
以parse-html为例：
import org.cyberneko.html.parsers.*;
这里报错是因为缺少 nekohtml-0.9.5.jar 如何获取nekohtml-0.9.5.jar：
到apache-nutch-1.7-bin/plugin 下搜索 nekohtml 就能找到这个jar包
然后复制到项目的lib文件夹里并add to build path 其他bug以此类推（所有的jar都可以在apache-nutch-1.7-bin/plugin 下找到 feed
cp apache-nutch-1.7-bin/plugins/feed/rome-0.9.jar apache-nutch-1.7-src/lib/
parse-html
cp apache-nutch-1.7-bin/plugins/parse-html/tagsoup-1.2.1.jar apache-nutch-1.7-src/lib/
cp apache-nutch-1.7-bin/plugins/lib-nekohtml/nekohtml-0.9.5.jar apache-nutch-1.7-src/lib/
至此整个工程将不会有任何错误了。 第五步：测试采集
1.vim conf/nutch-defalut.xml -----vim
/plugin.forlder ---vim查找命令
修该为：
<property>
<name>plugin.folders</name>
<value>./src/plugin</value>
<description>Directories where nutch plugins are located. Each
element may be a relative or absolute path. If absolute, it is used
as is. If relative, it is searched for on the classpath.</description>
</property>
原因：源代码文件中 plugin在src文件夹里，但在bin文件中plugin 在根目录下。
2 vim conf/nutch-site.xml 加入：
<property>
<name>http.agent.name</name>
<value>your sipder name</value>
</property>
3 在apache-nutch-1.7-src下建立一个urls文件夹，在urls下面建一个文本文档
mkdir urls
cd urls
vim seed.txt
写入：http://www.163.com/ 4 vim conf/regex-urlfilter.txt
5 运行配置：
运行结果：
至此运行成功。
检测采集结果：
统计结果：（unfetched比较多是因为nutch给url打分，过滤掉了分数小于0的，这个可以在nutch-default.xml中修改）
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - Statistics for CrawlDb: crawl/crawldb
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - TOTAL urls:
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - retry :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - min score: 0.0
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - avg score: 0.003186398
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - max score: 1.007
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - status (db_unfetched):
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - 3g..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - auto..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - baby..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - baoxian..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - bbs..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - bbs.culture..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - bbs.ent..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - bbs.lady..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - biz..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - blog..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - book..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - caipiao..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - cbachina..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - club.auto..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - corp..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - data.ent..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - digi..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - discovery..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - dl..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - ecard..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - edu..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - email..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - emarketing.biz..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - ent..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - expo..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - fashion..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - focus.news..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - fushi..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - game..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - gb.corp..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - hea..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - help..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - history.news..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - home..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - house..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - hr..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - jiu..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - kf.yxp..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - lady..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - live.caipiao..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - love..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - lovegongyi..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - m..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - media..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - mibao.gm..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - mobile..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - money..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - news..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - news.tag..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - newsapp.blog..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - pay..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - pic.auto..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - post.news..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - product.auto..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - qiye..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - quotes.money..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - reg..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - sports..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - survey2..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - t..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - tech..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - travel..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - tveasy.blog..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - v..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - v.money..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - v.news..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - v.sports..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - vipmail..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - vs.caipiao..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - wangyiyuedu.blog..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - war.news..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - www..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - yuedu..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - zx.caipiao..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - zz.yc..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - status (db_fetched):
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - caipiao..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - corp..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - digi..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - emarketing.biz..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - gb.corp..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - help..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - love..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - m..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - music..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - newsapp.blog..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - open..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - open.yuedu..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - sitemap..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - t..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - tech..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - www..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - yuedu..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - zz.yc..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - status (db_redir_temp):
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - 3g..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - m..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - status (db_redir_perm):
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - caipiao..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - corp..com :
-- ::, INFO crawl.CrawlDbReader (CrawlDbReader.java:processStatJob()) - CrawlDb statistics: done