lalbert / tokenizer-engine-for-wp
Search, related content and internal links for WordPress, with Elasticsearch-like text analysis and no search server.
Package info
github.com/lalbert/tokenizer-engine-for-wp
Type:wordpress-plugin
pkg:composer/lalbert/tokenizer-engine-for-wp
Requires
- php: >=8.2
- ext-mbstring: *
- composer/installers: ^1.0 || ^2.0
- lalbert/tokenizer-engine: ^1.3
- wamania/php-stemmer: ^4.0
Requires (Dev)
- friendsofphp/php-cs-fixer: ^3.64
- php-stubs/wordpress-stubs: ^6.5 || ^7.0
- phpstan/phpstan: ^2.1
- phpunit/phpunit: ^9.6
- szepeviktor/phpstan-wordpress: ^2.0
- wp-phpunit/wp-phpunit: ^6.5 || ^7.0
- yoast/phpunit-polyfills: ^2.0
Suggests
- ext-intl: Unicode word segmentation and ICU folding in the analyzers
Provides
None
Conflicts
None
Replaces
None
- dev-main
- 0.9.0
- 0.8.0
- 0.7.0
- 0.6.0
- 0.5.0
- 0.4.1
- 0.4.0
- 0.3.1
- 0.3.0
- 0.2.0
- 0.1.0
- dev-claude/eloquent-ride-x590w7
- dev-claude/bold-ptolemy-onbbj7
- dev-claude/feedback-0.8
- dev-claude/release-0.9.0
- dev-claude/elasticsearch-export
- dev-claude/incremental-related
- dev-claude/release-0.8.1
- dev-claude/release-0.8.0
- dev-claude/youthful-newton-q03v3q
- dev-claude/release-0.7.0
- dev-claude/did-you-mean
- dev-claude/internal-links-report
- dev-claude/seo-noindex
This package is auto-updated.
Last update: 2026-11-12 18:02:43 UTC
README
Search, related content and internal links for WordPress, with the text analysis of lalbert/tokenizer-engine (stemming, elisions, stop words per language, Elasticsearch-compatible analyzers). No search server needed.
Status: 0.9.
Français : README.fr.md (translation; this English documentation is the reference).
What it does
- Index the post types and taxonomies you choose, with the fields you choose (title, content, excerpt, taxonomies, custom fields) and a weight each. Shortcodes, blocks and synced patterns are cleaned so that the index holds what visitors read.
- Related content between any post types and taxonomies ("more like
this"), stable over time, with pinned and excluded items in the editor:
block "Related content", shortcode
[tokenizer_engine_related], PHP and REST APIs. - Search ranked with BM25 and highlighted, used for the search of the site (option) or through the APIs, with synonyms, the correction of typos ("Did you mean", when a search finds nothing) and the searches of visitors over 7, 30 or 90 days: the most frequent and the ones without results (screen Tools > Searches, option).
- Internal link suggestions in the block editor: passages mentioning the title of another post or term, linked in one click; suggested terms (categories, tags...) for a post.
- Automatic internal links (option): in the stored content (screen
Tools > Automatic links, API, WP-CLI, with a preview) or at display time, with guardrails (maximum per
content, one link per target, no link in headings, exclusions) and the
marker
data-te-link="auto". - Internal link counts: links in and out of each content in the lists of posts, and the orphan contents (no link in, in no menu), the ones with few links in and the dead ends (screen Tools > Internal links).
- Suggestions while typing in the search fields of the site (option).
- SEO plugins: the contents set to noindex in Yoast SEO, Rank Math, SEOPress, All in One SEO or The SEO Framework are left out of the results (option, on by default).
- Pages not found (404): the contents closest to the address requested,
suggested to the visitor without any redirect (option, block, shortcode
[tokenizer_engine_404]), and the list of the pages not found with their suggestions, to create redirects (screen Tools > Pages not found, option). - Multilingual sites with Polylang or WPML: each content analyzed in its language, search, related content and links within one language.
- Site Health: state of the index, background tasks late, size of the index (Tools > Site Health).
- A public API for developers: PHP, REST, WP-CLI and hooks.
Installation
With Composer, in a project that manages WordPress with Composer (Bedrock or
similar): the package has the type wordpress-plugin, so
composer/installers puts it in the
plugins directory.
composer require lalbert/tokenizer-engine-for-wp
Allow the plugin of composer/installers if Composer asks
(config.allow-plugins).
Without Composer for WordPress, clone the plugin and install its dependencies:
cd wp-content/plugins && git clone https://github.com/lalbert/tokenizer-engine-for-wp.git && cd tokenizer-engine-for-wp && composer install --no-dev
Then activate it, go to Settings > Tokenizer Engine (or the "Settings"
link of the plugin in the list of plugins), enable the post types
and taxonomies to index, and save: the index is built in the background
(WP-Cron), or now with wp tokenizer-engine-for-wp index rebuild.
WP-Cron runs on page views: on a site with little traffic, run it from a real
cron job (wp cron event run --due-now) so that the index and related
content stay up to date.
Sizes
The plugin is designed for up to about 10,000 indexed contents. After a
change, related content is recomputed for the sources it affects only (the
changed objects, the sources that list them and the closest ones); it is
recomputed for every indexed object every night, after a rebuild and beyond
200 grouped changes, which is the limit. Measures of benchmarks/scale.sh
(synthetic French posts of about 2 KB, PHP 8.3, MariaDB 10.11 with its
default settings, one core of a cloud VM):
| 5,000 posts | 20,000 posts | |
|---|---|---|
| Index rebuild | 42 s | 3 min |
| Size of the tables | 115 MB | 394 MB |
| Related content, recomputation after the change of one post | 2.4 s (57 sources) | 11 s (54 sources) |
| Related content, full recomputation (every night) | 2 min | 32 min |
| Search (median / p95) | 106 ms / 174 ms | 374 ms / 627 ms |
| Related content shown (stored) | 2 ms | 2 ms |
| Link suggestions of the editor | 238 ms / 284 ms | 1.9 s / 2.1 s |
| Suggestions of a page not found | 93 ms / 146 ms | 400 ms / 616 ms |
Beyond 10,000 contents, Tools > Site Health recommends leaving out the
post types that need no search or related content, or moving to a search
engine: wp tokenizer-engine-for-wp index export gives the settings of
Elasticsearch or OpenSearch indices that analyze like the plugin (see
Moving to Elasticsearch or OpenSearch). Run benchmarks/scale.sh 5000 to measure on your server (it
empties the database wordpress_bench, see the script).
Requirements: WordPress 6.5+, PHP 8.2+, MySQL or MariaDB. License: GPL-2.0-or-later.