Hi I want to share our concept to this problem here.
Our company, Wunderbyte, started building an AI agent as a new interface for our Booking module, but the approach turned out far too powerful for that one use case, so we're releasing it as a standalone plugin — a set of core skills plus a deliberately tiny, one-class skills API so third-party devs can plug in their own. It's already working; happy to hand out trials if you'd like to test, just contact me.
We implemented embeddings right from the start, because we want to support hundreds of skills and be fully language-agnostic, and neither a full skill catalog nor lexical search can do that . So we already had the building blocks for semantic matching in place — which led us to the site-search problem.
Core idea: keep the Search Areas, replace the engine.
Global Search cleanly separates the areas (\core_search\base subclasses that produce a document representation + access rules) from the engine (Solr/simpledb) that does keyword matching. We leave the areas untouched and swap only the matching layer, from keyword to vector similarity. Reusing the areas means we write zero per-plugin code: whitelisting a new area (a future Booking area, a third-party module) is a governance/config decision, not an agent-code change.
Concretely, per whitelisted area we:
- stream items incrementally with get_document_recordset($modifiedfrom, $context);
- build each item's document with the area's own get_document() — so we get its authoritative content assembly (content / title / contextid / courseid / owneruserid / modified) for free, including any custom logic the area applies;
- get a deep link back to the source with get_doc_url().
We then chunk + embed that content ourselves and retrieve by cosine similarity — true vector search, not the model guessing keywords.
Running the Area document pipeline without an active engine.
One wrinkle: an area's get_document() builds its document via document_factory, which resolves the current engine through manager::instance() — and that throws if no global-search engine is configured. We didn't want to force sites to turn on Global Search (extra config, a live search box, core cron indexing into simpledb we don't use). So the indexing task runs inside a short-lived, in-process engine session: we seed the search manager with a minimal in-memory engine for the duration of the task (in a try/finally), then tear it down. No $CFG change, no enableglobalsearch, no persisted state — it's memory-only and per-process, so parallel requests and other cron workers are completely unaffected. At query time we build the \core_search\document directly (passing our own engine to document_factory) for the deep links, so nothing touches the manager singleton in a live web request.
(This is the one part I'd genuinely like core devs' opinion on — is there a supported/cleaner way to run an area's get_document() "document-only", without standing up a full engine? It feels like a legitimate need for any "read the areas, bring your own matching" tool.)
Access — the hard part — comes for free.
We keep the index global and filter per user: pre-narrow to the user's allowed contexts, then gate every hit through the area's check_access() (ACCESS_GRANTED/DENIED/DELETED) as the sole authority. That's exactly Moodle's own model with Solr — no re-implemented permissions, no leaks. One catch we hit and solved: manager::get_areas_user_accesses() needs a configured engine, so for the pre-filter we replicate that context enumeration engine-free — and keep check_access() as the authority (the pre-filter is recall-only, never an allow-list).
This approach fulfills these demands:
1. Access for free (above) — the genuinely hard part, and we don't re-implement it.
2. No new mandatory plugin API. get_document() already is the content contract; an optional AI-specific content hook is reasonable but shouldn't be a prerequisite. Reusing the existing API makes every existing area indexable as-is and keeps maintenance sane.
3. We consume the area's document representation, not the tokenized search index — so we avoid the lossy clean-up the index applies and chunk the full content our way.
4. It runs without an active Solr/global-search engine (see the session above) — you don't have to become the site's search engine to get semantic retrieval.
5. External/global content fits the same store — our own documentation corpus lives in the same vector store next to course content.
Freshness/deletes: incremental via get_document_recordset($modifiedfrom) plus event observers that purge on context/module deletion.
And this leaves the door open to expose the same vector layer as a \core_search\engine later — so the native search box itself becomes semantic — without coupling the retrieval to it.
Very curious about your take — especially on the engine-session question above.
Georg
Our company, Wunderbyte, started building an AI agent as a new interface for our Booking module, but the approach turned out far too powerful for that one use case, so we're releasing it as a standalone plugin — a set of core skills plus a deliberately tiny, one-class skills API so third-party devs can plug in their own. It's already working; happy to hand out trials if you'd like to test, just contact me.
We implemented embeddings right from the start, because we want to support hundreds of skills and be fully language-agnostic, and neither a full skill catalog nor lexical search can do that . So we already had the building blocks for semantic matching in place — which led us to the site-search problem.
Core idea: keep the Search Areas, replace the engine.
Global Search cleanly separates the areas (\core_search\base subclasses that produce a document representation + access rules) from the engine (Solr/simpledb) that does keyword matching. We leave the areas untouched and swap only the matching layer, from keyword to vector similarity. Reusing the areas means we write zero per-plugin code: whitelisting a new area (a future Booking area, a third-party module) is a governance/config decision, not an agent-code change.
Concretely, per whitelisted area we:
- stream items incrementally with get_document_recordset($modifiedfrom, $context);
- build each item's document with the area's own get_document() — so we get its authoritative content assembly (content / title / contextid / courseid / owneruserid / modified) for free, including any custom logic the area applies;
- get a deep link back to the source with get_doc_url().
We then chunk + embed that content ourselves and retrieve by cosine similarity — true vector search, not the model guessing keywords.
Running the Area document pipeline without an active engine.
One wrinkle: an area's get_document() builds its document via document_factory, which resolves the current engine through manager::instance() — and that throws if no global-search engine is configured. We didn't want to force sites to turn on Global Search (extra config, a live search box, core cron indexing into simpledb we don't use). So the indexing task runs inside a short-lived, in-process engine session: we seed the search manager with a minimal in-memory engine for the duration of the task (in a try/finally), then tear it down. No $CFG change, no enableglobalsearch, no persisted state — it's memory-only and per-process, so parallel requests and other cron workers are completely unaffected. At query time we build the \core_search\document directly (passing our own engine to document_factory) for the deep links, so nothing touches the manager singleton in a live web request.
(This is the one part I'd genuinely like core devs' opinion on — is there a supported/cleaner way to run an area's get_document() "document-only", without standing up a full engine? It feels like a legitimate need for any "read the areas, bring your own matching" tool.)
Access — the hard part — comes for free.
We keep the index global and filter per user: pre-narrow to the user's allowed contexts, then gate every hit through the area's check_access() (ACCESS_GRANTED/DENIED/DELETED) as the sole authority. That's exactly Moodle's own model with Solr — no re-implemented permissions, no leaks. One catch we hit and solved: manager::get_areas_user_accesses() needs a configured engine, so for the pre-filter we replicate that context enumeration engine-free — and keep check_access() as the authority (the pre-filter is recall-only, never an allow-list).
This approach fulfills these demands:
1. Access for free (above) — the genuinely hard part, and we don't re-implement it.
2. No new mandatory plugin API. get_document() already is the content contract; an optional AI-specific content hook is reasonable but shouldn't be a prerequisite. Reusing the existing API makes every existing area indexable as-is and keeps maintenance sane.
3. We consume the area's document representation, not the tokenized search index — so we avoid the lossy clean-up the index applies and chunk the full content our way.
4. It runs without an active Solr/global-search engine (see the session above) — you don't have to become the site's search engine to get semantic retrieval.
5. External/global content fits the same store — our own documentation corpus lives in the same vector store next to course content.
Freshness/deletes: incremental via get_document_recordset($modifiedfrom) plus event observers that purge on context/module deletion.
And this leaves the door open to expose the same vector layer as a \core_search\engine later — so the native search box itself becomes semantic — without coupling the retrieval to it.
Very curious about your take — especially on the engine-session question above.
Georg