1.说明
- 爬虫采用Java的Jsoup
- ElasticSearch请在阿里云官网购买，采用客户端x-pack-transport
- 歌词网站来源：http://www.kuwo.cn/artist/index
2. 歌词网站分析
此处采用了比较笨的一种方式，即逐个分析每个请求的url，这样可以方便代码编写，就不用模拟器了（如需使用模拟器可参考使用cdp4j模拟点击事件等，但简单试了下不是很好用，且效率低）
a) 歌手获取分析
在http://www.kuwo.cn/artist/index 查看分页按钮的click事件，从js中找到分页请求的url
从click事件的artist.js中找到相关url如下图所示
其中pn参数即为页码参数
var b = host + "/artist/indexAjax?category=" + index + "&prefix=" + $("#artistContent").attr("data-letter") + "&pn=" + pn;如http://www.kuwo.cn/artist/indexAjax?category=0&prefix=&pn=5
返回结果如下图所示，即可获取各个歌手的链接，取得每个歌手名字的链接url即为歌手详情页。
接下来代码中只要循环遍历即可。
b)歌词分页获取分析
歌手详情页如：http://www.kuwo.cn/artist/content?name=%E5%88%98%E7%8F%82%E7%9F%A3
同样查看分页按钮click事件的js相关代码，可以找到如下url，以获取分页歌词信息。
如：http://www.kuwo.cn//artist/contentMusicsAjax?artistId=2&pn=1&rn=100
其中artistId为歌手id，pn为分页参数
接下来循环遍历即可
c)歌词获取分析
每首歌歌词的详情页，从上面已经可以循环遍历到，详情页如下，
跟进html元素获取到相应歌词信息即可
注意：此处有时因为无版权，无法显示歌词，需要增加异常处理
3. 项目配置
maven配置如下：
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.lewis</groupId>
<artifactId>crawlerToES</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.10.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.10.0</version>
</dependency>
<dependency>
<groupId>io.webfolder</groupId>
<artifactId>cdp4j</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>x-pack-transport</artifactId>
<version>5.3.3</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<optional>true</optional>
<version>5.3.3</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>4. 爬虫解析代码
a)获取歌手
public static List<String> getSingers() {
String url = host + "/artist/indexAjax?category=0&prefix=&pn=%d";
List<String> singer_urls = new ArrayList<String>();
int i = 0;
int max_len = 10000;
String last = null;
while (true) {
String cur_url = String.format(url, i);
try {
if(i>=max_len){
break;
}
Document doc = Jsoup.connect(cur_url).get();
String cur = doc.text();
if (cur.equals(last)) {
break;
}
Elements a_name = doc.getElementsByClass("a_name");
for (int i1 = 0; i1 < a_name.size(); i1++) {
Element a = a_name.get(i1);
String singer_url = a.attr("href");
System.out.println(singer_url);
singer_urls.add(singer_url);
}
last = cur;
i++;
} catch (IOException e) {
e.printStackTrace();
}
}
return singer_urls;
}b)获取某歌手的所有歌曲
public static List<String> getSongs(String singer_url) {
Document doc = null;
List<String> song_urls = new ArrayList<String>();
try {
doc = Jsoup.connect(host + singer_url).get();
String artistid = doc.getElementsByClass("artistTop").get(0).attr("data-artistid");
String url = "/artist/contentMusicsAjax?artistId=%s&pn=%d&rn=100";
int i = 0;
String last = null;
while (true) {
String cur_url = String.format(url, artistid, i);
doc = Jsoup.connect(host + cur_url).get();
String cur = doc.text();
if (cur.equals(last)) {
break;
}
Elements songs = doc.getElementsByClass("listMusic").get(0).children();
for (int j = 0; j < songs.size(); j++) {
Element song = songs.get(j);
String name = song.getElementsByClass("name").get(0).text();
String href = song.getElementsByClass("name").get(0).getElementsByTag("a").get(0).attr("href");
// System.out.println(name+":"+href);
song_urls.add(href);
}
i++;
last = cur;
}
} catch (IOException e) {
e.printStackTrace();
} finally {
return song_urls;
}
}c)获取某歌曲的歌词
public static SongBean getSong(String song_url) {
try {
Document doc = Jsoup.connect(host+song_url).get();
Element div_song = doc.getElementById("musiclrc");
// System.out.println(host+song_url);
if(div_song==null){
logger.info(song_url + "无法获取歌曲内容，可能是无版权!");
return null;
}
String song_name = div_song.getElementById("lrcName").text();
String singer = div_song.getElementsByClass("artist").get(0).getElementsByTag("span").get(0).getElementsByTag("a").get(0).text();
String album = div_song.getElementsByClass("album").get(0).getElementsByTag("span").get(0).getElementsByTag("a").get(0).text();
StringBuffer lyric = new StringBuffer();
if(div_song.getElementById("llrcId")!=null) {
Elements lyric_ps = div_song.getElementById("llrcId").children();
for (int i = 0; i < lyric_ps.size(); i++) {
lyric.append(lyric_ps.get(i).text() + "\n");
}
}else{
lyric.append("暂无歌词");
}
SongBean song = new SongBean(song_name,host+song_url,lyric.toString(),singer,album);
// System.out.println(String.format("%s:%s",song.getName(),song.getSinger()));
return song;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}5. 写入es
a)创建ElasticSearch Client
public static TransportClient openClient(String host, String clusterName, String securityUser){
Settings settings = null;
if (StringUtils.isNotBlank(securityUser)) {
settings = Settings.builder()
.put("cluster.name", clusterName)
.put("xpack.security.user",securityUser)
.build();
} else {
settings = Settings.builder()
.put("cluster.name", clusterName).build();
}
TransportClient transportClient = new PreBuiltXPackTransportClient(settings);
String ip = host.split(":")[0].trim();
int port = Integer.parseInt(host.split(":")[1].trim());
InetAddress add = null;
try {
add = InetAddress.getByName(ip);
} catch (UnknownHostException e) {
e.printStackTrace();
}
transportClient.addTransportAddress(new InetSocketTransportAddress(add, port));
return transportClient;
}b)批量写入elasticsearch
public static void writeToES(List<SongBean> list) throws Exception {
BulkRequestBuilder blukBuilder = client.prepareBulk();
for (SongBean song : list) {
IndexRequestBuilder indexBuilder = client.prepareIndex(es_indexName,es_indexType);
HashMap<String,String> map = new HashMap<>();
map.put("name",song.getName());
map.put("singer",song.getSinger());
map.put("album",song.getAlbum());
map.put("href",song.getHref());
map.put("lyric",song.getLyric());
indexBuilder.setSource(map);
indexBuilder.setTimeout("5d");
blukBuilder.add(indexBuilder);
}
BulkResponse bulkResponse = blukBuilder.execute().actionGet();
logger.info(String.format("write to elasticsearch songs count = %d",list.size()));
if (bulkResponse.hasFailures()) {
throw new Exception(bulkResponse.buildFailureMessage());
}
}6. 结果验证
爬取完歌曲数据后，登录阿里云ElasticSearch控制台，在Kibana控制台中，使用Dev Tools查询爬取到的歌曲数据，如下：
POST songs/sample/_search
{
"query": {
"match_all": {}
}
}