Database

[kj7rrv]

Because the models are getting large, we should probably store them in a database. Probably one table for logs (storing text and category) and another for the compiled model? Maybe we should just compile the model every time CS it starts? It doesn't seem to take too long, and doing that would make it easier to load a new model (just restart CedarSentinel). We might have to stop CS when retraining if we use a DB; do you think that will be a problem? I guess since CS doesn't read logs while running, we could have the model editor remove the logs from the DB (or mark them as "being processed" in a new DB column)? @fire219

[fire219]

We're at the point we definitely need to consider using a proper database. Storing logs and model(s) in it is logical.

I don't see having to compile at startup being a major problem (GPTC already has to do that if you forgot to manually compile it).

Not sure why would have have to stop CS while retraining? Most database systems are designed for simultaneous transactions, so data corruption shouldn't be an issue. This will also be nice when we implement automatic training for known good messages.

I see no issues with deleting logged messages once they are processed.

[kj7rrv]

Yeah, now that I think about it, we shouldn't have to stop the bot. If we use numerical keys for each message, we should be able to modify rows without affecting CS's ongoing logging. Do you think we should have a separate thread that recompiles the model periodically when it has been changed, so we don't have to restart the bot?

[fire219]

Once we have a need for it, that'll be a good idea. But that shouldn't be a priority; we need to get the database functionality working at a core level first, and that thread won't really do anything until we have automatic training (or functionality for moderators to tag a message in-chat).

In a few minutes I should be pushing up the v0.6 update, which should bring automatic spam deletion (when paired with an unfortunately-not-upstream Matterbridge build)

[kj7rrv]

Okay sounds good. Automatic deletion will be nice. Is there any chance of upstreaming the matterbridge changes?

[fire219]

The main Matterbridge devs seem to be entirely uninterested. The request has been sitting over there (42wim/matterbridge#1737) for 15 days now.

[kj7rrv]

Okay. Maybe they would accept a PR?

[kj7rrv]

What DB do you think we should use? Probably SQLite?

[42wim]

The main Matterbridge devs seem to be entirely uninterested. The request has been sitting over there (42wim/matterbridge#1737) for 15 days now.

The matterbridge devs is 1 single dev that has a limited amount of time

[kj7rrv]

Done in #36