Support for on-device or self-hosted models

[Pr0methean]

This would be a lot cheaper to use, with more privacy, if we could run it with a self-hosted LLM instead of relying on one of the corporate providers. Could you please add support for that?

[FlorianHeigl]

I think this will also matter - or especially - for journalists in news orgs when they have some confidential data someone gave them, it would put the person sharing at risk if the data can't be siloed to be only on their LLM host.
same for many researchers who license data sets and need to keep them close to the heart because they're not allowed to share them.
while outgoing agent calls and queries to public records can still give a public of what my imagined journalist is investigating, it would still obfuscate who he is and avoid leaking out the data he was given.

So just for use cases that really matter in terms of humanity and whatever, and aren't just personal preferences.

My personal side is just I have a server with a GPU and don't wanna pay someone else for the same thing.
OTOH I have more specifically a servr with an AMD GPU so you could also say I will end up paying someone else for a working thing. But you get the idea :>

[Pr0methean]

The important thing from my perspective is that for me to use this right now, I'd have to commit to a monthly spend with one of the AI providers; and I'm currently unemployed, which means nobody has committed to a regular spend on me.