首先安装
pip install selenium
测试抓取baidu,其中的chromedriver.exe需要自己下载,百度有很多的
import time
from selenium import webdriver browser = webdriver.Chrome('C:\Program Files (x86)\Google\ChromeDriver\chromedriver.exe') # 可选参数,如果不指定将搜索环境变量
browser.get('http://www.baidu.com/')
time.sleep(5)
执行结果如下
进阶
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys driver = webdriver.Chrome('C:\Program Files (x86)\Google\ChromeDriver\chromedriver.exe')
driver.get('https://www.baidu.com/')
elem = driver.find_element_by_id('kw')
elem.send_keys("test")
elem.send_keys(Keys.RETURN)
print(driver.page_source) #<input type="text" name="passwd" id="passwd-id" /> element = driver.find_element_by_id("passwd-id")
element = driver.find_element_by_name("passwd")
element = driver.find_elements_by_tag_name("input")
element = driver.find_element_by_xpath("//input[@id='passwd-id']")
如果遇到错误
C:\Users\bin\AppData\Local\Programs\Python\Python36\python.exe C:/Users/bin/PycharmProjects/Test/Test/Crawler/TestSelenium.py
Traceback (most recent call last):
File "C:\Users\bin\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\common\service.py", line 74, in start
stdout=self.log_file, stderr=self.log_file)
File "C:\Users\bin\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 707, in __init__
restore_signals, start_new_session)
File "C:\Users\bin\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 990, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] 系统找不到指定的文件。 During handling of the above exception, another exception occurred: Traceback (most recent call last):
File "C:/Users/bin/PycharmProjects/Test/Test/Crawler/TestSelenium.py", line 3, in <module>
browser = webdriver.Chrome()
File "C:\Users\bin\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 62, in __init__
self.service.start()
File "C:\Users\bin\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\common\service.py", line 81, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
原因是没有安装chrome driver,到 https://sites.google.com/a/chromium.org/chromedriver/downloads 下载,
如果是windows就将chrome driver添加到Path环境变量中,Linux直接放到 usr/bin目录下