Addition of other OpenAi compatible models

[Ammar-create]

Is your feature request related to a problem? Please describe.
Yes. Right now the app only works with one AI model provider. That is kind of limiting. Some people want to use other OpenAI-compatible models because they might be cheaper, faster, or just better for certain tasks. It feels frustrating to be stuck with only one option when there are many models out there that could work the same way.
Describe the solution you'd like
I would like the app to let users connect other OpenAI-compatible API models. Maybe in the settings there could be:
A place to paste a custom API base URL
A place to enter an API key
A way to choose which model to use from that provider
This would make the app more flexible and useful for more people. Users could choose the model they like instead of being forced to use only one.
Describe alternatives you've considered
I thought about using a proxy server to connect to other models, but that sounds complicated and not very practical for most users. Another option is waiting for the app to officially support more providers, but that could take a long time. It would be better if users could just connect any OpenAI-compatible API themselves.
Additional context
A lot of AI apps now support custom OpenAI-compatible endpoints, and it makes them more powerful. Adding this feature would make the app more future-proof and more attractive to people who like to customize their tools. It would also help students and developers who want to experiment with different models.

[Kishlay-notabot]

Adding support for a smaller model (qwen3.5 0.9b) would be great for even more resource constrained phones. Please take a look at that! the realtime webGPU demos are quite fun and nice for a model of such size.

[AnsahMohammad]

Thanks for the suggestion. I'll defenitely take a look and try to bring few more models in the next update.

Qwen sounds promising, not sure if it can be used with the existing APIs or not.

[Kishlay-notabot]

hi @AnsahMohammad
I recently came across your project while brainstorming an idea that’s very similar to what you’ve built.I was thinking about creating a kind of “second brain” for the phone, not just limited to OCR and LLM-based tagging of screenshots, but also incorporating voice notes with on-device speech-to-text. For example, there are very fast STT models like MoonshineAI (~25 MB for the English model) that can run locally. My idea was to have a unified searchable space where both screenshot content and transcribed audio notes live together, so I could query across them and retrieve relevant information.
I originally planned to build this myself, but your project already implements much of what I had in mind. I was wondering if you’d be open to expanding the feature scope, and whether I could explore the codebase and contribute. One thing I’d be interested in working on is adding an on-device STT pipeline so audio notes could be indexed alongside screenshot text.

I’m not primarily an app developer (also that I don't want to totally vibe code it irresponsibly.), but I could collaborate with friends who are stronger on the app side. I’d mainly focus on the model/inference side and integration.
Would love to get your thoughts on this idea.

[AnsahMohammad]

hi @AnsahMohammad
I recently came across your project while brainstorming an idea that’s very similar to what you’ve built.I was thinking about creating a kind of “second brain” for the phone, not just limited to OCR and LLM-based tagging of screenshots, but also incorporating voice notes with on-device speech-to-text. For example, there are very fast STT models like MoonshineAI (~25 MB for the English model) that can run locally. My idea was to have a unified searchable space where both screenshot content and transcribed audio notes live together, so I could query across them and retrieve relevant information.
I originally planned to build this myself, but your project already implements much of what I had in mind. I was wondering if you’d be open to expanding the feature scope, and whether I could explore the codebase and contribute. One thing I’d be interested in working on is adding an on-device STT pipeline so audio notes could be indexed alongside screenshot text.

I’m not primarily an app developer (also that I don't want to totally vibe code it irresponsibly.), but I could collaborate with friends who are stronger on the app side. I’d mainly focus on the model/inference side and integration.
Would love to get your thoughts on this idea.

Second brain sounds really good. Its a great idea. I'm assuming it's something similar to Nothing's Essential space. That I think would require a separate app (or some serious refactor) as currently we aren't able to directly modify the screenshot just as a user takes it, unless user opens the app and decides to manually record the note.
To clarify few things;
Do you want the ability to add audio alone ? Or will all the audio notes be always linked to a screenshot ?

There was a plan in the roadmap to allow audio notes for screenshot in the "add a note" section. Was deprioritized as users could directly use native keyboard STT.

Is your request adding a simple STT for users to voice type into the note section? Or do you also have plans to do some features like essential space?

[Kishlay-notabot]

hey! thanks for the quick reply. For my version of the idea i don't really want to link it to a specific screenshot, it could be something like a quick homescreen widget like nothing probably has and we could speak into it for just noting things down, and that data gets stored at a unified location where the screenshot annotations are also stored. And not sure about the screenshot tagging thing because I believe then we would have to be deep into the android's system to have such perms right? So yeah I was thinking of just having a simple STT feature, then the possibilities of this app easily increases by a lot! And yes I had nothing's essential space in mind but I didn't want to specifically mention it. Because yeah if we do implement it it won't directly be apples to apples due to the permission limitations and how deep can we go into the stack.
About the nothing's essential space feature, i think the button takes a screenshot and pops up an audio recording option where we talk and maybe ask the AI to schedule things for us etc.
This being an open source project, I understand the hardware constraints because I want this to be totally offline and secure, without worrying about internet or API costs.

[AnsahMohammad]

Sounds good, but if it involves audio recording without screenshot, then i don't think it will directly fit this app's design or target users. I can extend shots studio to bring the STT in the next update for "add a note" section for screenshots.

But that being sad, i'll be very open to building/supporting an open source fully local essential-space app, but may not be perfect, but can work with what's possible without the deep 1st party integration. Basically an app, which can store the screenshots/audios/notes, which user can query later, or suggest tasks, calender etc.

I saw your work on MonkeSearch, A similar RAG will be a great addition to this app, enhancing the image search. But not sure how much resources it consumes/requires. Let me know your thoughts on this.

[Kishlay-notabot]

I agree to your point that the audio note taking doesn't really fit the user base of this specific app, but maybe if I fork this and we develop a prototype just as a PoC? Audio + screenshots, and they both getting summarized by super small llms, possibly by the same one which does the image scanning. At the end we can still achieve the idea of "second brain" by having independent audio STT feature and your existing implementation, without having to link audios to screenshots.

Or maybe if we keep an independent audio recording feature and use a vector db, content with similar context would be indexed closer so what if we leave the linking to the vector db? and avoid manual linking as a whole?

A screenshot of a "food recipe website" would be closely indexed to an audio recording of me talking about a random "food recipe".

And yes, talking about monkesearch, I recently saw qdrant's edge version where they open sourced a very light implementation of their vector db tool. Maybe we could look into that, and then if we dump both the screen shot summaries and audio summaries to the same db, we can query anything and find all of them at one place. I'm also not sure about the resource usage here, would have to dig into it a bit.
How does your current search mechanism work? Do the images have some textual summary metadata attached to them? ( Like "screenshot containing face of 3 people" etc.) Are you using vector dbs of any kind till now for the search? If yes then it would be a bit easier to implement these features I guess? I haven't brainstormed super hard and went into the details about how would the unified system work. So yeah sorry if I am leaving gaps in the idea, I'm just thinking out loud.

And on a side note, monkesearch's initial idea never included vector dbs, i was trying to have an LLM create specific queries for the specified OS's existing filesystem index, but have a natural language bridge for it.

[AnsahMohammad]

I'd prefer using a fork rather than independently matching to the screenshot as there can be lot of gap, and another headache to match these to the right image.

A simple fork prototype with support for audio can be done. But beware that app currently doesnt use vector db anywhere and we'll have to start migrating the storage to vector for LLM based searches to work. I don;t have experience working with vector dbs, but i assume is something implementable with minor changes.

To give context on the current implemetation; Every image object has the filename, ai-description, tags, and notes. We currently use shared_preferences to store data. There is no database as such.
For adding the audio, simply extend the object with a new field for audio_note should do the job (atleast for the prototype), this way it can be directly made searchable in the current system. ( basically an audio note will be a screenshot without image but an audio transcript).

Now it depends on the scope of "second brain" that you are trying to implement. WHat exactly is it supposed to do other than making your audio searchable ?
I'm curious on what the end goal plan here is? are you thinking of helping users query their ideas to be fetched later ? does it also involve suggestions ? or is there even a larger scope.

---

And on a side note, monkesearch's initial idea never included vector dbs, i was trying to have an LLM create specific queries for the specified OS's existing filesystem index, but have a natural language bridge for it.

The hacky way to bridge the file query is very clever, nice one ggs.

[Kishlay-notabot]

Oh okay so you mean you want to go the way where we can associate audios with screenshots right? What I meant with fork was to maybe spin up a whole different project with your code as base. But it's fine.

Now it depends on the scope of "second brain" that you are trying to implement. WHat exactly is it supposed to do other than making your audio searchable ?

Coming back to what I thought of, I mean the crux of my idea was to have a note taking app which has smart screenshot monitoring (your whole project) and another feature where i could just simply tap and record, and both of the stuff gets stored in the knowledge base for reference. I am not looking too far away towards doing tool calls etc. to make the app actually do things about it (schedule stuff etc.) because

1. We are resource constrained in terms of hardware and I am constrained by my lack of understanding of the android filesystem and how the entire operating system works.
2. That thing is just essentially a VERY large project. And me not being from android dev background would be just like talking in thin air about it.

So yeah, my plan was just to have a simple note taking app for the sake of smart organization of things i want to remember and keep it for reference.
Which way do you think we should go?

The hacky way to bridge the file query is very clever, nice one ggs.

Thanks haha. Quite a fun project it was and I still aim to work on it much more, but just constrained with some stuff for now.

[thara-hangnung]

Is your feature request related to a problem? Please describe.
Yes. Right now the app only works with one AI model provider. That is kind of limiting. Some people want to use other OpenAI-compatible models because they might be cheaper, faster, or just better for certain tasks. It feels frustrating to be stuck with only one option when there are many models out there that could work the same way.
Describe the solution you'd like
I would like the app to let users connect other OpenAI-compatible API models. Maybe in the settings there could be:
A place to paste a custom API base URL
A place to enter an API key
A way to choose which model to use from that provider
This would make the app more flexible and useful for more people. Users could choose the model they like instead of being forced to use only one.
Describe alternatives you've considered
I thought about using a proxy server to connect to other models, but that sounds complicated and not very practical for most users. Another option is waiting for the app to officially support more providers, but that could take a long time. It would be better if users could just connect any OpenAI-compatible API themselves.
Additional context
A lot of AI apps now support custom OpenAI-compatible endpoints, and it makes them more powerful. Adding this feature would make the app more future-proof and more attractive to people who like to customize their tools. It would also help students and developers who want to experiment with different models.

I'm no developer and I don't know how hard it is to implement such features.
But Yes. This must be implemented ASAP.

[AnsahMohammad]

I'm no developer and I don't know how hard it is to implement such features.
But Yes. This must be implemented ASAP.

Hey,
Can understand your frustration. We are bringing support for ollama in the next update, and will rollout support for other models in coming updates.
Thank you