前不久。看到一篇文章我用爬虫一天时间“偷了”知乎一百万用户。仅仅为证明PHP是世界上最好的语言，该文章中使用的登录方式是直接复制cookie到代码中，这里呢，我不以爬信息为目的。仅仅是简单的介绍使用java来进行模拟登录的基本过程。之前写过的文章android 项目实战——打造超级课程表一键提取课表功能事实上就是模拟登录的范畴。再加上近期在知乎上看到非常多人问关于超级课程表的实现，事实上本质就是模拟登录，掌握了这篇文章的内容，你不再操心抓不到信息了。然后，这篇文章会使用到之前的一篇Cookie保持的文章Android OkHttp的Cookie自己主动化管理，还有Jsoup的使用Jsoup库使用全然解析，为了简单处理，直接使用javaSE来，而不再使用Android进行。假设要移植到Android，唯一的处理可能就是把网络请求工作扔到子线程中去 。
首先使用Chrome打开知乎首页 , 点击登录，你会看到以下这个界面
在Chorme中按F12，调出开发人员工具，切到Network选项卡，勾选Preserve Log。记得一定要勾选，不然你会看不到信息。
一切就绪后，在输入框中输出账号密码点击登录。登录成功后你会看到这么一条记录
点击图中的email，在最下方你会看到本次请求提交了4个參数，以及在上方，你会看到本次请求的地址是http://www.zhihu.com/login/email
你会吃惊的发现知乎的密码是明文传输的，提交的參数的意思也非常easy，email就是账号，password就是密码。remember_me就是是否记住，这里传true就能够了，另一个_xsrf參数，这个毛估估应该是防爬虫的。
因此在提交前我们要从源代码中将这个值抓取下来。该值在表单的隐藏域中
一切准备就绪后，你就兴高採烈的用代码去模拟登录，然后你会发现会返回一个验证码错误的信息。事实上，我们还须要提交一个验证码，其參数名为captcha，验证码的地址为，
http://www.zhihu.com/captcha.gif?r=时间戳于是我们得出了这种一个数据。
- 请求地址
http://www.zhihu.com/login/email- 请求參数
_xsrf 表单中提取的隐藏域的值
captcha 验证码
email 邮箱
password 密码
remember_me 记住我另一个问题。验证码的值怎么得到呢。答案是人工输入。将验证码保存到本地进行觉得识别，输入后进行登陆就可以。
这里的网络请求使用OkHttp。以及解析使用Jsoup，然后我们会使用到Gson，将他们增加maven依赖
com.squareup.okhttp
okhttp
2.4.0
org.jsoup
jsoup
1.8.3
com.google.code.gson
gson
2.3.1
" data-snippet-id="ext.954a1d7d6c4276c4faf06435bd4518df" data-snippet-saved="false" data-csrftoken="20AinXcm-dRsprSXazhrt6nBpMoaeJ3OOLFA" data-codota-status="done"> <dependencies>
<dependency>
<groupId>com.squareup.okhttp</groupId>
<artifactId>okhttp</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.8.3</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.3.1</version>
</dependency>
</dependencies>
在编码之前。我们得想想怎么维持登陆状态。没错，就是Cookie怎样保持，我们仅仅进行登陆一次，兴许都直接採集数据就能够了，因此须要将cookie持久化。对之前的文章中的一个Android类进行改造。使其变成java平台可用的类。能够看到我们将它从之前保存到SharePrefrences中改成了保存到文件里，并以json形式存储，这就是为什么会用到Gson的原因了
package cn.edu.zafu.zhihu;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.reflect.TypeToken;
import java.io.*;
import java.net.CookieStore;
import java.net.HttpCookie;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
/**
* User:lizhangqu(513163535@qq.com)
* Date:2015-07-18
* Time: 16:54
*/
public class PersistentCookieStore implements CookieStore {
private static final Gson gson= new GsonBuilder().setPrettyPrinting().create();
private static final String LOG_TAG = "PersistentCookieStore";
private static final String COOKIE_PREFS = "CookiePrefsFile";
private static final String COOKIE_NAME_PREFIX = "cookie_";
private final HashMap<String, ConcurrentHashMap<String, HttpCookie>> cookies;
private Map<String,String> cookiePrefs=new HashMap<String, String>();
/**
* Construct a persistent cookie store.
*
*/
public PersistentCookieStore() {
String cookieJson = readFile("cookie.json");
Map<String,String> fromJson = gson.fromJson(cookieJson,new TypeToken<Map<String, String>>() {}.getType());
if(fromJson!=null){
System.out.println(fromJson);
cookiePrefs=fromJson;
}
cookies = new HashMap<String, ConcurrentHashMap<String, HttpCookie>>();
// Load any previously stored cookies into the store
for(Map.Entry<String, ?> entry : cookiePrefs.entrySet()) {
if (((String)entry.getValue()) != null && !((String)entry.getValue()).startsWith(COOKIE_NAME_PREFIX)) {
String[] cookieNames = split((String) entry.getValue(), ",");
for (String name : cookieNames) {
String encodedCookie = cookiePrefs.get(COOKIE_NAME_PREFIX + name);
if (encodedCookie != null) {
HttpCookie decodedCookie = decodeCookie(encodedCookie);
if (decodedCookie != null) {
if(!cookies.containsKey(entry.getKey()))
cookies.put(entry.getKey(), new ConcurrentHashMap<String, HttpCookie>());
cookies.get(entry.getKey()).put(name, decodedCookie);
}
}
}
}
}
}
public void add(URI uri, HttpCookie cookie) {
String name = getCookieToken(uri, cookie);
// Save cookie into local store, or remove if expired
if (!cookie.hasExpired()) {
if(!cookies.containsKey(uri.getHost()))
cookies.put(uri.getHost(), new ConcurrentHashMap<String, HttpCookie>());
cookies.get(uri.getHost()).put(name, cookie);
} else {
if(cookies.containsKey(uri.toString()))
cookies.get(uri.getHost()).remove(name);
}
cookiePrefs.put(uri.getHost(), join(",", cookies.get(uri.getHost()).keySet()));
cookiePrefs.put(COOKIE_NAME_PREFIX + name, encodeCookie(new SerializableHttpCookie(cookie)));
String json=gson.toJson(cookiePrefs);
saveFile(json.getBytes(), "cookie.json");
}
protected String getCookieToken(URI uri, HttpCookie cookie) {
return cookie.getName() + cookie.getDomain();
}
public List<HttpCookie> get(URI uri) {
ArrayList<HttpCookie> ret = new ArrayList<HttpCookie>();
if(cookies.containsKey(uri.getHost()))
ret.addAll(cookies.get(uri.getHost()).values());
return ret;
}
public boolean removeAll() {
cookiePrefs.clear();
cookies.clear();
return true;
}
public boolean remove(URI uri, HttpCookie cookie) {
String name = getCookieToken(uri, cookie);
if(cookies.containsKey(uri.getHost()) && cookies.get(uri.getHost()).containsKey(name)) {
cookies.get(uri.getHost()).remove(name);
if(cookiePrefs.containsKey(COOKIE_NAME_PREFIX + name)) {
cookiePrefs.remove(COOKIE_NAME_PREFIX + name);
}
cookiePrefs.put(uri.getHost(), join(",", cookies.get(uri.getHost()).keySet()));
return true;
} else {
return false;
}
}
public List<HttpCookie> getCookies() {
ArrayList<HttpCookie> ret = new ArrayList<HttpCookie>();
for (String key : cookies.keySet())
ret.addAll(cookies.get(key).values());
return ret;
}
public List<URI> getURIs() {
ArrayList<URI> ret = new ArrayList<URI>();
for (String key : cookies.keySet())
try {
ret.add(new URI(key));
} catch (URISyntaxException e) {
e.printStackTrace();
}
return ret;
}
/**
* Serializes Cookie object into String
*
* @param cookie cookie to be encoded, can be null
* @return cookie encoded as String
*/
protected String encodeCookie(SerializableHttpCookie cookie) {
if (cookie == null)
return null;
ByteArrayOutputStream os = new ByteArrayOutputStream();
try {
ObjectOutputStream outputStream = new ObjectOutputStream(os);
outputStream.writeObject(cookie);
} catch (IOException e) {
System.out.println("IOException in encodeCookie"+ e);
return null;
}
return byteArrayToHexString(os.toByteArray());
}
/**
* Returns cookie decoded from cookie string
*
* @param cookieString string of cookie as returned from http request
* @return decoded cookie or null if exception occured
*/
protected HttpCookie decodeCookie(String cookieString) {
byte[] bytes = hexStringToByteArray(cookieString);
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
HttpCookie cookie = null;
try {
ObjectInputStream objectInputStream = new ObjectInputStream(byteArrayInputStream);
cookie = ((SerializableHttpCookie) objectInputStream.readObject()).getCookie();
} catch (IOException e) {
System.out.println("IOException in decodeCookie"+e);
} catch (ClassNotFoundException e) {
System.out.println("ClassNotFoundException in decodeCookie"+e);
}
return cookie;
}
/**
* Using some super basic byte array <-> hex conversions so we don't have to rely on any
* large Base64 libraries. Can be overridden if you like!
*
* @param bytes byte array to be converted
* @return string containing hex values
*/
protected String byteArrayToHexString(byte[] bytes) {
StringBuilder sb = new StringBuilder(bytes.length * 2);
for (byte element : bytes) {
int v = element & 0xff;
if (v < 16) {
sb.append('0');
}
sb.append(Integer.toHexString(v));
}
return sb.toString().toUpperCase(Locale.US);
}
/**
* Converts hex values from strings to byte arra
*
* @param hexString string of hex-encoded values
* @return decoded byte array
*/
protected byte[] hexStringToByteArray(String hexString) {
int len = hexString.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(hexString.charAt(i), 16) << 4) + Character.digit(hexString.charAt(i + 1), 16));
}
return data;
}
public static String join(CharSequence delimiter, Iterable tokens) {
StringBuilder sb = new StringBuilder();
boolean firstTime = true;
for (Object token: tokens) {
if (firstTime) {
firstTime = false;
} else {
sb.append(delimiter);
}
sb.append(token);
}
return sb.toString();
}
public static String[] split(String text, String expression) {
if (text.length() == 0) {
return new String[]{};
} else {
return text.split(expression, -1);
}
}
public static void saveFile(byte[] bfile, String fileName) {
BufferedOutputStream bos = null;
FileOutputStream fos = null;
File file = null;
try {
file = new File(fileName);
fos = new FileOutputStream(file);
bos = new BufferedOutputStream(fos);
bos.write(bfile);
} catch (Exception e) {
e.printStackTrace();
} finally {
if (bos != null) {
try {
bos.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}
if (fos != null) {
try {
fos.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
}
public static String readFile(String fileName) {
BufferedInputStream bis = null;
FileInputStream fis = null;
File file = null;
try {
file = new File(fileName);
fis = new FileInputStream(file);
bis = new BufferedInputStream(fis);
int available = bis.available();
byte[] bytes=new byte[available];
bis.read(bytes);
String str=new String(bytes);
return str;
} catch (Exception e) {
e.printStackTrace();
} finally {
if (bis != null) {
try {
bis.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}
if (fis != null) {
try {
fis.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
return "";
}
}然后新建一个OkHttp请求类，并设置其Cookie处理类为我们编写的类。
private static OkHttpClient client = new OkHttpClient();
client.setCookieHandler(new CookieManager(new PersistentCookieStore(), CookiePolicy.ACCEPT_ALL));好了。能够開始获取_xsrf以及验证码了。验证码保存在项目根文件夹下名为code.png的文件
private static String xsrf;
public static void getCode() throws IOException{
Request request = new Request.Builder()
.url("http://www.zhihu.com/")
.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")
.build();
Response response = client.newCall(request).execute();
String result = response.body().string();
Document parse = Jsoup.parse(result);
System.out.println(parse + "");
result = parse.select("input[type=hidden]").get(0).attr("value")
.trim();
xsrf=result;
System.out.println("_xsrf:" + result);
String codeUrl = "http://www.zhihu.com/captcha.gif?r=";
codeUrl += System.currentTimeMillis();
System.out.println("codeUrl:" + codeUrl);
Request getcode = new Request.Builder()
.url(codeUrl)
.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")
.build();
Response code = client.newCall(getcode).execute();
byte[] bytes = code.body().bytes();
saveCode(bytes, "code.png");
}
public static void saveCode(byte[] bfile, String fileName) {
BufferedOutputStream bos = null;
FileOutputStream fos = null;
File file = null;
try {
file = new File(fileName);
fos = new FileOutputStream(file);
bos = new BufferedOutputStream(fos);
bos.write(bfile);
} catch (Exception e) {
e.printStackTrace();
} finally {
if (bos != null) {
try {
bos.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}
if (fos != null) {
try {
fos.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
}然后将获取来的參数连同账号密码进行提交登录
public static void login(String randCode,String email,String password) throws IOException{
RequestBody formBody = new FormEncodingBuilder()
.add("_xsrf", xsrf)
.add("captcha", randCode)
.add("email", email)
.add("password", password)
.add("remember_me", "true")
.build();
Request login = new Request.Builder()
.url("http://www.zhihu.com/login/email")
.post(formBody)
.addHeader("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")
.build();
Response execute = client.newCall(login).execute();
System.out.println(decode(execute.body().string()));
}
public static String decode(String unicodeStr) {
if (unicodeStr == null) {
return null;
}
StringBuffer retBuf = new StringBuffer();
int maxLoop = unicodeStr.length();
for (int i = 0; i < maxLoop; i++) {
if (unicodeStr.charAt(i) == '\\') {
if ((i < maxLoop - 5)
&& ((unicodeStr.charAt(i + 1) == 'u') || (unicodeStr
.charAt(i + 1) == 'U')))
try {
retBuf.append((char) Integer.parseInt(
unicodeStr.substring(i + 2, i + 6), 16));
i += 5;
} catch (NumberFormatException localNumberFormatException) {
retBuf.append(unicodeStr.charAt(i));
}
else
retBuf.append(unicodeStr.charAt(i));
} else {
retBuf.append(unicodeStr.charAt(i));
}
}
return retBuf.toString();
}当看到以下的信息就代码登录成功了
之后你就能够获取你想要的信息了。这里简单获取一些信息，比方我要获取*哥的followers的昵称。分页自己处理下就ok了。
public static void getFollowers() throws IOException{
Request request = new Request.Builder()
.url("http://www.zhihu.com/people/zord-vczh/followees")
.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")
.build();
Response response = client.newCall(request).execute();
String result=response.body().string();
Document parse = Jsoup.parse(result);
Elements select = parse.select("div.zm-profile-card");
StringBuilder builder=new StringBuilder();
for (int i=0;i<select.size();i++){
Element element = select.get(i);
String name=element.select("h2").text();
System.out.println(name+"");
builder.append(name);
builder.append("\n");
}
}下图就是获取到的信息。当然。仅仅要你登录了。什么信息你都能够获取到。
最后上源代码,Intelij的maven项目
http://download.csdn.net/detail/sbsujjbcy/8984375