It’s not something we’ve figured out yet, so would be worth exchanging ideas around this.
We did co-fund this plugin https://github.com/catalyst/moodle-tool_dataflows that was meant to help integrate with Snowflake via Snowpipe but things changed and we’re now heading towards using Azure Data Lake.
From a data protection standpoint I’d really like to understand how we avoid having data from anonymous submissions being revealed to users who aren’t privileged to see it, once released in the lake.
And how we do this without replicating a bunch of application logic into the lake.
Or maybe there’s a common set of data cleansing rules to cover the application logic we could collaborate on?
It’s something we’re going to start properly exploring from next week until DACH moot so should be great to bounce ideas
We did co-fund this plugin https://github.com/catalyst/moodle-tool_dataflows that was meant to help integrate with Snowflake via Snowpipe but things changed and we’re now heading towards using Azure Data Lake.
From a data protection standpoint I’d really like to understand how we avoid having data from anonymous submissions being revealed to users who aren’t privileged to see it, once released in the lake.
And how we do this without replicating a bunch of application logic into the lake.
Or maybe there’s a common set of data cleansing rules to cover the application logic we could collaborate on?
It’s something we’re going to start properly exploring from next week until DACH moot so should be great to bounce ideas