I propose the creation of Moodle code that makes it easy to input drawing e.g. handwriting or drawing and for that to be processed by an external AI. In combination with audio, this could open up Moodle to pre-literate children. For example e.g. audio that says “draw a flower”, and the prompt asks if the drawing is anything like a flower and to generate a text or audio response.
I have experimented with the Question bank questiongen plugin as updated by
ISB Bayern - ByCS Lernplattform
https://github.com/bycs-lp/moodle-qbank_questiongen
And was very impressed with how AI/LLM systems can extract images from text and I also experimented with using AI to extract text from my own hand writing. I have looked at various javascript “signature” libraries that seem to do a good job of taking stylus (or similar) input.
My specific use case is my AI Text plugin for maths symbols, I can also imaging the STACK question type being adapted for it. I was also thinking of it as a generic library to be used by other plugins.
The processing would be controlled by a configurable prompt, for example maths symbols could be converted to LayTex and then undergo additional processing.This would be at its best with Stylus input of course. The code could take the form of a new form element e.g.
$mform->addElement(‘imageforai’,’calculationanswer’,’Calculation Answer’);
Or perhaps it could be a TinyMCE button?
