-
Download package and paste the package in your laravel app.
-
Add the package folder path in composer's repositories section as local, and update your require section. For example:
"require" : {
"nxvhm/newscraper": "dev-master"
},
"repositories": {
"local": {
"type": "path",
"url": "app/Library/nxvhm/newscraper"
}
},
-
Run
composer require "nxvhm/newscraper dev-master". This will symlink the package to the vendor/ folder and install its dependencies and treat it as regular composer pkg. At this point the package should be auto-discoverable from laravel. -
Publish config and migration:
php artisan vendor:publish --provider="Nxvhm\Newscraper\NewscraperServiceProvider"
- Run the migration
php artisan migrate
Register strategies in database in order for them to have unique id
php artisan newscraper:register-sites
Start from artisan with the following cmd:
php artisan scrape:news {StrategyName}
Where strategy name is an existing strategy class.
To create scraping strategy class via CLI:
php artisan newscraper:create-strategy
After creating it, you should update the autoload maps and register it in db:
composer dump-autoload
php artisan newscraper:register-sites
For customized logic on saving article to db provide the responsible class in the config file in custom_save. The class should implement the Nxvhm\Newscraper\Contracts\ArticleSaver contract. Example:
<?php
namespace App;
use Nxvhm\Newscraper\Strategies\Strategy;
use Nxvhm\Newscraper\Contracts\ArticleSaver as ArticleSaverInterface;
use Illuminate\Support\MessageBag;
class ArticleSaver implements ArticleSaverInterface
{
public static function saveArticle(array $article, Strategy $strategy): MessageBag
{
# Custom logic goes here
dd($article, $strategy->name);
}
}
- Define site strategies from a config file
- --Allow strategy class lookup in a configurable namespaces not only in a single one--
- Implement more strategies
- Implement mechanism for Flexible Time/Date scraping