All of us data hoarders get to a point where we need to circumvent in-place rules that prevent us from scraping the web. Enter: Selenium. The world’s favorite tool for automating tasks in a browser. Selenium uses ChromeDriver, initially created for automated testing, but also a splendid scraping tool. In this blog post, n00b stuff.
Let’s solve a basic Python issue regarding Selenium:
"MESSAGE: 'CHROMEDRIVER' EXECUTABLE NEEDS TO BE IN PATH"
In essence, your chromedriver executable cannot be found, because its not registered in the PATH. To fix this, there are multiple things you can do.
Fix 1: Manually specify the path to chromedriver.exe
You can easily provide the link to chromedriver.exe as a string to the Chrome() method.
from selenium import webdriver chrome_driver = webdriver.Chrome('C:\path\to\chromedriver.exe')
Fix 2: move chromedriver.exe to your workspace
By saving chromedriver.exe in the same folder als your Python working directory, there’s no need to specify the path.
Fix 3: add the directory of chromedriver.exe to your PATH variable
Adding directories to the PATH variable can be done in multiple ways:
- This blog post explains how you can do it via the Windows UI
- This blog post explains how you can do it in the terminal on Windows, Mac and Linux