PHP Scraper

🏢 (09-Apr-2022) See your PHP code do web browsing on your screen with GUI. Laravel 13.

• Important note:

• Using Postman?

• Requirements

• Installation

• Usage

• Adding a new command

• Browser Testing

• Contributing

• License

Important note:

Before you try to scrape any website, go through its robots.txt file. You can access it via domainname/robots.txt. There, you will see a list of pages allowed and disallowed for scraping. You should not violate any terms of service of any website you scrape.

Using Postman?

Postman client.

Published Postman API Collection.

Requirements

• https://laravel.com/docs
• Java

Installation

cp .env.example .env
# Don't worry when the following step errors related to chromedriver binary, we will install them right after.
composer install

Add chromedriver to Path

Make sure Chromedriver is installed and added to your environment Path.

# install chromedriver for Panther client.
vendor/bin/bdi detect drivers
sudo mv drivers/chromedriver /usr/local/bin/chromedriver
# Or
# chromedriver_mac64
# chromedriver_win32
# See https://chromedriver.storage.googleapis.com
# for drivers list.
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/local/bin/chromedriver
chromedriver --version

Continue installation

composer install
php artisan key:generate
# Before running the next command:
# Update your database details in .env
php artisan migrate --seed
yarn install
yarn build

Download Selenium Server jar file

Download Selenium Server jar file.

Run the following in a new terminal.

java -jar selenium-server-[version].jar standalone --override-max-sessions true --max-sessions 10

CLI options in the Selenium Grid.

Usage

Update the command at ./app/Console/Commands/BrowserScrape.php

php artisan browser:scrape

BrowserInvoker.php

Panther Environment Variables

Panther Environment Variables.

Capabilities

Capabilities.

Using Desired Capabilities.

Adding a new command

php artisan make:crawler TestCrawler
``

## Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

## License
[BSD](https://opensource.org/licenses/BSD-3-Clause)