omnicrawl / scrapingbee
Omnicrawl's ScrapingBee, under any PHP host: pages fetched with or without JavaScript, the target's status, headers and final address reported, behind omnicrawl's gate - and only for targets declared under a licence (one's own site, written permission), because every request goes through ScrapingBee
Requires
- php: >=8.2
- omnisense/contracts: ^1.0
- omnisense/omnicrawl: ^1.0
- symfony/http-client: ^6.4|^7.0|^8.0
Requires (Dev)
- phpstan/phpstan: ^2.1
- phpunit/phpunit: ^11.0
Suggests
None
Provides
None
Conflicts
None
Replaces
None
README
ScrapingBee for omnisense/omnicrawl, under any PHP host. Pages with or without JavaScript (GET /api/v1), the target's status, headers and final address reported, the family's User-Agent forwarded, no premium or stealth proxy, no retry through another proxy - for targets declared as one's own site or with written permission only, because every request goes through ScrapingBee's rotating proxies, which hide who is collecting.
It is reached through omnicrawl's gate only: every address is admitted first (the target's declaration, robots.txt read by the core, TDM reservations, the host's limits), the service is called after, and what it brings back is checked again and made a record with its provenance. Read what omnicrawl refuses to do, and why.
composer require omnisense/omnicrawl omnicrawl/scrapingbee symfony/http-client
It needs a ScrapingBee account and its API key (the api_key option); its tests run on recorded answers and need
none. A first call with no account: omnicrawl's README (the recorded client, or its built-in direct provider).
$registry = new Registry([new ScrapingBeeFactory($http)], [ 'scrapingbee' => ['factory' => 'scrapingbee', 'options' => ['api_key' => 'SCRAPINGBEE_API_KEY']], ], $gate);
The key goes by reference: SCRAPINGBEE_API_KEY names an environment variable (or a secret of the
SecretsInterface given to the factory), never the key itself. Its usage policy is
policy.json (limits, terms, what they say of lawful use), read on the day it says; the limits of each plan the service publishes are its plans, picked with the plan option (hobby (default), freelance, startup, business, business-plus, enterprise-14, -24, -41, -69, -120). See
docs/index.md.
Tested on recorded answers written from the service's documentation; no test calls the service.
MIT licence.