Search by

lalbert / tokenizer-engine-for-wp

lalbert

Search, related content and internal links for WordPress, with Elasticsearch-like text analysis and no search server.

Package info

github.com/lalbert/tokenizer-engine-for-wp

Type:wordpress-plugin

pkg:composer/lalbert/tokenizer-engine-for-wp

Statistics

Installs: 42

Dependents: 0

Suggesters: 0

Stars: 0

Open Issues: 29


README

Search, related content and internal links for WordPress, with the text analysis of lalbert/tokenizer-engine (stemming, elisions, stop words per language, Elasticsearch-compatible analyzers). No search server needed.

Status: 0.9.

Français : README.fr.md (translation; this English documentation is the reference).

What it does

  • Index the post types and taxonomies you choose, with the fields you choose (title, content, excerpt, taxonomies, custom fields) and a weight each. Shortcodes, blocks and synced patterns are cleaned so that the index holds what visitors read.
  • Related content between any post types and taxonomies ("more like this"), stable over time, with pinned and excluded items in the editor: block "Related content", shortcode [tokenizer_engine_related], PHP and REST APIs.
  • Search ranked with BM25 and highlighted, used for the search of the site (option) or through the APIs, with synonyms, the correction of typos ("Did you mean", when a search finds nothing) and the searches of visitors over 7, 30 or 90 days: the most frequent and the ones without results (screen Tools > Searches, option).
  • Internal link suggestions in the block editor: passages mentioning the title of another post or term, linked in one click; suggested terms (categories, tags...) for a post.
  • Automatic internal links (option): in the stored content (screen Tools > Automatic links, API, WP-CLI, with a preview) or at display time, with guardrails (maximum per content, one link per target, no link in headings, exclusions) and the marker data-te-link="auto".
  • Internal link counts: links in and out of each content in the lists of posts, and the orphan contents (no link in, in no menu), the ones with few links in and the dead ends (screen Tools > Internal links).
  • Suggestions while typing in the search fields of the site (option).
  • SEO plugins: the contents set to noindex in Yoast SEO, Rank Math, SEOPress, All in One SEO or The SEO Framework are left out of the results (option, on by default).
  • Pages not found (404): the contents closest to the address requested, suggested to the visitor without any redirect (option, block, shortcode [tokenizer_engine_404]), and the list of the pages not found with their suggestions, to create redirects (screen Tools > Pages not found, option).
  • Multilingual sites with Polylang or WPML: each content analyzed in its language, search, related content and links within one language.
  • Site Health: state of the index, background tasks late, size of the index (Tools > Site Health).
  • A public API for developers: PHP, REST, WP-CLI and hooks.

Installation

With Composer, in a project that manages WordPress with Composer (Bedrock or similar): the package has the type wordpress-plugin, so composer/installers puts it in the plugins directory.

composer require lalbert/tokenizer-engine-for-wp

Allow the plugin of composer/installers if Composer asks (config.allow-plugins).

Without Composer for WordPress, clone the plugin and install its dependencies:

cd wp-content/plugins && git clone https://github.com/lalbert/tokenizer-engine-for-wp.git && cd tokenizer-engine-for-wp && composer install --no-dev

Then activate it, go to Settings > Tokenizer Engine (or the "Settings" link of the plugin in the list of plugins), enable the post types and taxonomies to index, and save: the index is built in the background (WP-Cron), or now with wp tokenizer-engine-for-wp index rebuild.

WP-Cron runs on page views: on a site with little traffic, run it from a real cron job (wp cron event run --due-now) so that the index and related content stay up to date.

Sizes

The plugin is designed for up to about 10,000 indexed contents. After a change, related content is recomputed for the sources it affects only (the changed objects, the sources that list them and the closest ones); it is recomputed for every indexed object every night, after a rebuild and beyond 200 grouped changes, which is the limit. Measures of benchmarks/scale.sh (synthetic French posts of about 2 KB, PHP 8.3, MariaDB 10.11 with its default settings, one core of a cloud VM):

5,000 posts 20,000 posts
Index rebuild 42 s 3 min
Size of the tables 115 MB 394 MB
Related content, recomputation after the change of one post 2.4 s (57 sources) 11 s (54 sources)
Related content, full recomputation (every night) 2 min 32 min
Search (median / p95) 106 ms / 174 ms 374 ms / 627 ms
Related content shown (stored) 2 ms 2 ms
Link suggestions of the editor 238 ms / 284 ms 1.9 s / 2.1 s
Suggestions of a page not found 93 ms / 146 ms 400 ms / 616 ms

Beyond 10,000 contents, Tools > Site Health recommends leaving out the post types that need no search or related content, or moving to a search engine: wp tokenizer-engine-for-wp index export gives the settings of Elasticsearch or OpenSearch indices that analyze like the plugin (see Moving to Elasticsearch or OpenSearch). Run benchmarks/scale.sh 5000 to measure on your server (it empties the database wordpress_bench, see the script).

Requirements: WordPress 6.5+, PHP 8.2+, MySQL or MariaDB. License: GPL-2.0-or-later.