selenium 和 phantomJS或chrome浏览器抓取渲染网页

首先pip安装selenium

一、phantomjs

1、下载phantomjs压缩包,解压,把bin文件夹路径加入PATH环境变量

2、代码

#coding=utf-8
import requests
import  re
from pyquery import PyQuery as pq
from lxml import etree
from bs4 import BeautifulSoup
import sys
from selenium import webdriver
reload(sys)
sys.setdefaultencoding("utf-8")


def getHtml(url):
    driver = webdriver.PhantomJS(executable_path='/home/lhy/phantomjs-1.9.8-linux-x86_64/bin/phantomjs')
    driver.get(url)
    fo = open("phonesinfo2.txt", "wb")
    fo.write(driver.page_source)
    fo.close()
    return driver.page_source


二、chrome浏览器

1、必须安装chrome浏览器

2、下载chrome驱动chromedriver

3、把驱动加如PATH环境变量(注意最好修改/etc/profile配置,永久生效)

4、代码

#coding=utf-8
import requests
import  re
from pyquery import PyQuery as pq
from lxml import etree
from bs4 import BeautifulSoup
import sys
from selenium import webdriver
reload(sys)
sys.setdefaultencoding("utf-8")


def getHtml(url):
    driver=webdriver.Chrome();
    driver.get(url)
    fo = open("phonesinfo2.txt", "wb")
    fo.write(driver.page_source)
    fo.close()
    return driver.page_source
注意运行过程中会打开chrome浏览器



上一篇:Windows 蠕虫首次现身 Mac 电脑:伪装成 Adobe Flash 软件


下一篇:浅谈js中startsWith函数不能在任何浏览器兼容的问题