Full RAG integration in local_ai_manager and - partially - moodle in general

Full RAG integration in local_ai_manager and - partially - moodle in general

von Philipp Memmel -
Anzahl Antworten: 12

Hi everyone,

the anticipation is growing beyond measure! Can't wait to head to Zürich!

So here's my pitch:

Introduction

Some of you might have heard of the AI plugins bundle which we (the team of the "ByCS Lernplattform" - formely known as "mebis") developed:

What we're still missing, but is heavily requested from our user base, is having course content as context in the AI utils, especially the AI chat. The qbank_questiongen plugin already does that in some way, but it's not really satisfactory. What really would be beneficial is having "access" to the course content (and even more, "global" content) and be able to do this in a way that does not just bloat the context of the model by inserting the extracted text of all the 100 PDFs with 100 pages each into the context window of the external LLM.

RAG functionality has already been discussed in this forum and there are already existing proof of concepts or even stable plugins that are doing some kind of RAG. However, we want the RAG to be deeply integrated in both our tools, be configured in the same user interface where the configuration in the AI manager plugin is already happening, and - and that is currently an open issue - into moodle.

So here comes the pitching part:
  • Take the work that Michael Hughes already did as proof of concept as starting point (
  • Integrate it in a way that it fits into the current local_ai_manager architecture
  • Develop a frontend extension for our AI chat plugin so teachers can configure what content should be indexed by the RAG system
  • Develop a general Moodle API for plugins (this would be a core contribution), so plugins itself can control and eventually provide the content to a RAG indexing system, because that's what we're really lacking currently.
Some additional explanation on the moodle contribution part:
  • RAG means that content is being indexed and stored in an external vector store.
  • But which data is "THE" data of a plugin / activity instance?
  • For example, a "lesson" activity has some internal database tables. Some of the information is usable for RAG, some is not. Each plugin has to decide what content it exposes and at what conditions.
  • So there is the need of a Moodle API to allow (activity) plugins to provide this content whenever AI wants to access content (does not matter if for RAG purposes or just for direct context retrieving).
What we're hoping to have at the end of the dev camp:
  • A chatbot (we already do have that) in which you can define the context that is being added from the RAG system (we do not have that part yet)
  • A user interface in which you can select which activities of a course should be indexed.
  • A backend system that is indexing/embedding the selected activities.
  • Chat with an AI model that generates responses based on a lookup of information from the previously filled vector store
  • If we got a lot of time and developer capabilities: Maybe also have global RAG content (moodle docs etc.) which can also be used in the chatbot.
What we need
  • (Backend) Developers
  • Frontend designers
  • Testers and "users"
  • Lots of coffee
  • The MoodlemootDACH spirit to make it happen!

If you are interested, join the group, you do not need to have any prior experience!

Looking forward to see you in Zürich!

Philipp

Tags:
Als Antwort auf Philipp Memmel

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von Julia Lee -
I’m in too. Thank you for taking the initiative, Philipp!
Als Antwort auf Philipp Memmel

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von Georg Maißer -

Just out of curiosity: Wouldn't it be a possibility to use the Moodle search API for the RAG integration?

Als Antwort auf Georg Maißer

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von Michael Hughes -

That has been the basis for my past research, and it only goes so far.

I'd explored extending this, but it's very much geared up towards "search" rather than RAG. We extended the solr engine try to piggy back on the indexing aspect, and got to a reasonable point (https://github.com/mhughes2k/moodle-search_solrrag)

The ideal route would be to extend the Moodle Search infrastructure but this really needs the coordination with Moodle Core.

Als Antwort auf Michael Hughes

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von Michael Hughes -
Also some of the search indexing performs "cleaning up" of content, which impacts the quality of the resource for RAG / AI, as the structure of the content is also lost.

There is also the existing search infrastructure that delegates the "extraction" or generation of a representation of a given activity into an appropriate format, which would be really good to leverage.
Als Antwort auf Michael Hughes

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von David Herney -
Hi...

What about a hybrid approach? I implemented a test case where AI extracts keywords from each user question to then perform a content search within Moodle. The AI ​​then answers the question using the search results (including links to the original resource). With this strategy, there is no need to share all the platform content with the AI, and it can be implemented quickly without always relying on vector databases. Furthermore, this strategy allows for the use of other "service-based" flow controls, for example: answering questions about activity dates or badge acquisition (information lookups that don't depend on the content itself).

The major advantage is that security controls are already in place, and the search API is already functional.

You may have already tried this method. I’d like to know if it works well or if there are significant drawbacks. I can share my proof-of-concept if it would be useful for testing purposes.

Saludos
Als Antwort auf David Herney

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von Philipp Memmel -

Hi David,

thank you very much for sharing this idea. The approach sounds promising and low-level and I have not thought of that possibility.

One downside that immediately comes to my mind: We not only want course content to be available, but also external content which can be made available in the vector store. Not sure if one would find a way of doing this with your approach as well?

Besides that I can imagine (it's nothing more than just (educated) guessing so far) that your approach could be a bit unreliable. If I got you right, you are generating keywords and the success of the search will depend on "luck" if the AI generated the ones that match the search results. A vector similarity search should be (much?) more reliable I could imagine?

Also, I'm still not convinced that a plugin really wants to provide the same data to a search as to a vector store, so an additional API for providing content to AI systems would make sense to me.

Any more thoughts?

Regards,

Philipp

Als Antwort auf Philipp Memmel

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von David Herney -

Hi Philipp...

"also external content which can be made available in the vector store": What other content? We create site-level resources when we want the information to be available to everyone. Moodle offers many resources for doing this while maintaining control.

However, we create "knowledge bases" containing new content specifically for AI responses. In those cases, the source isn't displayed as a standard resource; it's used solely to generate answers. We did create vectors in these instances, but the results weren't great—I believe it was an implementation issue.

Yes, having all that content available might make the AI ​​more precise via vector search, but that doesn't reflect the reality of Moodle. The first issue is, obviously, the risks associated with information access. The second is scope. In an educational context, you shouldn't answer a question about a definition using just *any* correct content; context is crucial. You might have two courses that discuss "numbers," but the way the topic is handled differs vastly between 10-year-olds and master's degree students. Consequently, the answer to "What is a number?" shouldn't be the same; it likely depends on the specific course. If a course doesn't actually cover the topic of numbers, should the AI ​​answer the question at all? When answers are based on course material, I believe the response should rely exclusively on that course's content and always include a reference to the full source material.

We initially used keywords for searches, but wildcards and other strategies can be employed to find optimal results. That would offer the best of both worlds.

From a technical standpoint, adding more APIs to plugins makes maintenance increasingly complex. Sticking to current APIs makes life easier for developers. Furthermore, the same API could be used to implement a vector indexing layer to enhance search capabilities if needed—but as an optional feature, not a mandatory one.

Lots to talk about... that's great, I like this.

Saludos

Als Antwort auf David Herney

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von Michael Hughes -
This is almost closer to Andrej Karparthy's idea of a wiki-llm, where there is an intermediary "knowledge base", curated by an AI, which is consulted first, rather than going direct to the source.

However, if the Search API you're referring to is the Moodle Global Search API, this in itself is already performing an indexing task, using the activity modules to provide the correct "representation" for textual search, which is then stored in a search backend.

In this case the search back end is optimised for text search (words), rather than similarity search (vectors).

So yes this would work, but the context that is returned would be constrained by the ability of the AI to form appropriate text queries. This is also similar to a model in RAG where the "original user query" is not the one used to retrieve results, but used to synthesise a number of variants of the query, and the multiple result sets being amalgamated into a (hopefully) better set.

The index of content in the "search engine" (text or vector) is also always going to be constrained by it's "fresh-ness", which means that storing anything in there that is potentially dynamic is not a great idea. The Moodle Search API does then do some further checks against the actual Moodle database (e.g. checking user-visibility) etc.

This was also one of the reasons that I was originally keen to leverage the Global Search API in implementing the solr_rag engine, so that a lot of these security and contextual checks we'd effectively "get for free". This also meant that the vector and text indexes were in the same backend... but this potentially means you might get a less flexible set of capabilities for both requirements.

In both cases some form of "processing" needs to be done, to generate the text-search index or the vector for a piece of content, and in this 2nd aspect, yes the content has to be passed to a text-embedding AI model, but this is also different from the LLM model being used against the results.

I still feel that the Global Search API is the "right" area to look at in terms of the indexing and generation of a search index, but it has some limitations due to being primarily geared towards "text" search.

M
Als Antwort auf Philipp Memmel

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von David Pesce -
This sounds interesting!
Als Antwort auf Philipp Memmel

Re: Full RAG integration in local_ai_manager and - partially - moodle in general

von Wolfgang Sandholzer -
If my pitch doesn't find developers I could be a tester/user and coffee drinker