omnisense / omnicrawl
Omnicrawl: fetch and extract web pages under any PHP host, through crawling services or directly, responsible by default - the target site's rules applied by the family itself before any service is called. robots.txt read and honoured per RFC 9309 for the family's own token, a usage policy per targe
Requires
- php: >=8.2
- ext-dom: *
- ext-libxml: *
- ext-mbstring: *
- omnisense/contracts: ^1.0
- symfony/http-client-contracts: ^3.0
Requires (Dev)
- phpstan/phpstan: ^2.1
- phpunit/phpunit: ^11.0
- symfony/http-client: ^6.4|^7.0|^8.0
Suggests
- omnicrawl/apify: Apify: runs over a set of addresses with the Cheerio Scraper actor, raw HTML and headers kept
- omnicrawl/firecrawl: Firecrawl (v2): pages fetched, rendered or not, and batch runs over a set of addresses
- omnicrawl/scrapingbee: ScrapingBee: pages fetched, rendered or not, the target's headers reported, for targets declared under a licence only (its rotating proxies hide who collects)
- omnicrawl/zyte: Zyte API: pages fetched over HTTP, for targets declared under a licence only (its ban avoidance cannot be turned off)
- omnisense/omnibridge: The host contracts; omnibridge/symfony and omnibridge/laravel give a shared limiter and store
- symfony/http-client: The HTTP client the gate reads robots.txt with and the direct provider fetches with (any HttpClientInterface does)
Provides
None
Conflicts
None
Replaces
None
README
Web pages fetched and extracted under any PHP host, through crawling services or directly, responsible by default: the target site's rules are applied by the family itself, before any service is called. One contract by capability, one package per provider, as every family of the Omni ecosystem, on omnisense/contracts.
- robots.txt honoured, read by the core with the family's User-Agent and applied per RFC 9309; no option to ignore it, an override only for one's own site or with written permission, with a written reason.
- A policy per target site, declared as data: allowed or refused, on what authority (own site, written permission, published terms, robots.txt only), with a reference and a date, its rate limits, what may be stored and republished. A site nobody declared gets the conservative default: single pages for previews, one request every 10 seconds and 100 a day per host, no rendering, no crawl run. Each application declares the sites it refuses (the family ships no list of third-party sites).
- An identified User-Agent with a contact, required by every factory; per-host limits beside the provider's own, the stricter winning.
- X-Robots-Tag, meta robots and TDMRep reservations read and honoured.
- Provenance on every record: source, fetch time, provider, authority. Records are not open data.
- No evasion: none of the services' block-bypassing features is exposed.
Contracts: page-fetch, extract (metadata, Open Graph, Twitter cards, JSON-LD, selectors),
crawl-runs. Providers: a built-in direct fetcher, and
omnicrawl/firecrawl,
omnicrawl/scrapingbee,
omnicrawl/zyte,
omnicrawl/apify.
Installation and a first call
composer require omnisense/omnicrawl
composer require symfony/http-client # the client the gate reads robots.txt with, and the direct provider fetches with
No account and no network for this first call: the recorded client of the Testing kit (omnisense/contracts'
RecordedHttpClient) answers for the site, robots.txt included. The built-in direct provider needs no account
either, only a real HTTP client and a User-Agent naming a contact; Firecrawl, ScrapingBee, Zyte and Apify each
need an account and its API key (omnicrawl/<provider>, api_key option), and ScrapingBee and Zyte a target
declared as one's own site or with written permission.
require __DIR__.'/vendor/autoload.php'; use Omnicrawl\{Agent, Registry}; use Omnicrawl\Direct\DirectFactory; use Omnicrawl\Guard\Gate; use Omnicrawl\Testing\RecordedHttpClient; $http = RecordedHttpClient::routes([ // the site's answers, recorded: symfony/http-client's HttpClient::create() in production '~^GET https://news\.example\.com/robots\.txt$~' => [200, "User-agent: *\nDisallow: /drafts/\n", ['content-type' => ['text/plain']]], '~^GET https://news\.example\.com/articles/1$~' => [200, '<html><head><title>Article</title><meta property="og:title" content="An article"></head></html>', ['content-type' => ['text/html']]], ]); $gate = new Gate(Agent::parse('ExampleBot/1.0 (+https://www.example.org/bot; bot@example.org)'), $http); $registry = new Registry([new DirectFactory($http)], ['preview' => ['factory' => 'direct']], $gate); echo $registry->extract('preview')->extract('https://news.example.com/articles/1')->preview()['title'], "\n"; // An article
With a real client, the targets the application declared, and a link preview:
$gate = new Gate(Agent::parse('ExampleBot/1.0 (+https://www.example.org/bot; bot@example.org)'), $http, Targets::fromFile('targets.json')); $registry = new Registry([new DirectFactory($http)], ['preview' => ['factory' => 'direct']], $gate); $registry->extract('preview')->extract('https://news.example.com/story')->preview();
Read what omnicrawl refuses to do, and why, then how to declare a target and the rest.
MIT licence.