README

TYPO3 extension with console commands to crawl websites, generate BackstopJS scenarios and verify legacy URLs/redirects after a relaunch.

Commands

crawler:crawl

vendor/bin/typo3 crawler:crawl [concurrency=25] [delay=0] \
  [--url=www.example.org] [--output-dir=tests/crawls/legacy] [--track-redirects]

• Without --url the domain is asked interactively (default: DDEV_HOSTNAME).
• Logs are written pipe-separated to tests/crawls/<timestamp>/full_logs.csv and <status>_logs.csv (message|status|url|found_on_url|content_type).
• --track-redirects adds the columns redirect_chain (301 https://… > 301 https://…) and final_url. Without it, redirects are followed silently and the final status is logged for the original URL.

Legacy URL check (relaunch / redirect verification)

Goal: after a relaunch, know for every URL of the old website whether it still works (same), is redirected (redirected), returns 404/410 (not_found) or is broken (error).

1. Build the inventory once (old website, usually still live)

vendor/bin/typo3 crawler:legacy-inventory \
  --sitemap=https://www.example.org/sitemap.xml \
  --sitemap=https://www.example.org/en/sitemap.xml \
  --output-dir=tests/redirect-check/legacy

• Sitemap indexes are followed recursively; image:loc etc. are ignored.
• --crawl-log=tests/crawls/<ts>/full_logs.csv (repeatable) adds URLs of a crawler:crawl run against the old website (2xx + text/html rows and 3xx rows). Useful for pages not listed in any sitemap.
• --host= restricts the inventory to hosts (default: hosts of the given sitemaps).
• Sitemaps are not exhaustive: add historic sources using repeatable --url-list=old-urls.txt (one absolute HTTP(S) URL per line, blank lines and # comments allowed).
• Status checks are off by default. Opt in with --check-status to store original status, final URL and every redirect hop.
• Trailing-slash variants remain separate. Fragments and host case are deduplicated.
• Malformed/unavailable sitemaps abort the entire inventory operation. Empty results are rejected. Writes are atomic; an existing inventory requires explicit --force.
• Result: inventory.csv (url,source,legacy_status,legacy_category,legacy_final_url,legacy_notes,legacy_redirect_chain). Existing six-column inventories remain readable. Keep this file so comparisons do not fetch the old site again.

2. Compare against the new system (any number of times)

vendor/bin/typo3 crawler:compare-legacy \
  --inventory=tests/redirect-check/legacy/inventory.csv \
  --target-base=https://staging.example.org

• Scheme/host of each legacy URL is replaced with --target-base (default: DDEV primary URL); path and query are kept.
• Redirects are followed manually (--max-redirects, default 10), loops are detected.
• All inventory URLs are checked, including those that already redirected on the legacy site (they must keep redirecting). --only-direct restricts the check to recorded direct-2xx legacy URLs.
• Options: --concurrency, --timeout, --auth=user:password, --insecure, --fail-on-problems (exit code 1 on review_needed/not_found/error), --output-dir (default tests/redirect-check/reports/<timestamp>).
• --timeout and --insecure apply to a dedicated HTTP client, not to TYPO3's global HTTP settings.
• Basic Auth is sent only to the original target origin (scheme, host and effective port), never to another origin or an HTTPS-to-HTTP downgrade. Embedded URL credentials are rejected.
• Output: report.csv (every redirect status and destination, initial and final URL/status), report.md (concise summary), plus console summary. CSV rows are flushed on completion, preserving completed results on interruption; Markdown is written only after a completed run. Use a new output directory for each run; existing report.csv files are not overwritten.
• Per-URL transport/malformed redirect errors produce error rows and do not abort other checks. Configuration errors return exit 2, input/output failures exit 1; completed runs normally exit 0. Concurrency and timeout must be positive; max-redirects must be a nonnegative integer.
• This checks HTTP reachability, not semantic destination correctness. Without an expected target mapping, even a same-origin permanent redirect to 2xx is not proof of the correct page.

Notes: chain_<n> (several hops), temporary_<status> (302/303/307 instead of 301/308), to_homepage (deep URL ends on a (language) start page), external_origin (any hop leaves the target origin, including staging → old live), expected_redirect_missing (recorded legacy redirect now answers directly with 2xx), status_<status>. Redirect evidence requires --check-status or a previously checked inventory; historical redirect behaviour cannot be inferred from a sitemap alone.

Tip: on DDEV pages are rendered uncached - the first run is slow; a second run is much faster.

Development

composer install
composer test    # PHPUnit
composer stan    # PHPStan level 8
composer cs      # PHP-CS-Fixer (PER-CS 2.0), dry run; composer cs:fix applies it