(2022) Use PHP technologies to crawl and click buttons on websites with GUI. I highly recommend working with Linux (including virtual machines) or MacOs.
Before you try to scrape any website, go through its robots.txt file. You can access it via domainname/robots.txt. There, you will see a list of pages allowed and disallowed for scraping. You should not violate any terms of service of any website you scrape.
cp .env.example .env
# Don't worry when the following step errors related to chromedriver binary, we will install them right after.
composer installMake sure Chromedriver is installed and added to your environment Path.
# install chromedriver for Panther client.
vendor/bin/bdi detect drivers
sudo mv drivers/chromedriver /usr/local/bin/chromedriver
# Or
# chromedriver_mac64
# chromedriver_win32
# See https://chromedriver.storage.googleapis.com
# for drivers list.
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/local/bin/chromedriver
chromedriver --versioncomposer install
php artisan key:generate
# Before running the next command:
# Update your database details in .env
php artisan migrate --seed
npm install
npm run buildDownload Selenium Server jar file.
Run the following in a new terminal.
java -jar selenium-server-4.18.1.jar standalone --override-max-sessions true --max-sessions 10CLI options in the Selenium Grid.
Update the command at ./app/Console/Commands/BrowserScrape.php
php artisan browser:scrapePanther Environment Variables.
php artisan make:crawler TestCrawlerSee Python Selenium web scraper.
See Github to Bitbucket Backup Repo Updater.
See food nutrition facts search web app.
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.