docs: No information how to stop the inference

[onzag]

What was unclear or otherwise insufficient?

I am checking all over the docs as how can I stop the inference during the onToken function as tokens are generated. I have a very particular usecase where I can tell, after certain amount of tokens if it's going correctly or I should stop the generation.

Most cases result in unhandled exceptions, calling completion.dispose(); seems to be a terrible idea as it results in unhandled exceptions that don't seem to matter where my try and catch are, right now I throw an error onToken and then call completition.dispose on the error handling, but it hangs like crazy; like I may as well just had left the competition end because it seems that it takes as much.

Like I'd like to see in documentation how it is done, it should be in the default example, in the tutorial, but it doesn't seem to be there; it is nowhere specified, throwing an error seems to be the best option, and completition.dispose seems to fail more often than not, throw an error and then calling content dispose seems to be the only way to prevent the unhandled exception, but it's really slow way, so it can't be that, it'll be faster to kill the program and reload the model than this function.

Sorry but I've spent too much and still haven't been able to reliably stop the inference, if you tell me how, I don't mind making a PR for docs, should be simple enough of a doc PR.

Recommended Fix

Just put the couple of lines in the documentation how to stop the inference.

Additional Context

No response

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, and I know how to start.

[giladgd]

The docs contain an example on how to stop a generation, you just have to pass an abort signal.
All the APIs behave in the same manner and you can pass an abort signal to cancel an ongoing operation.

[onzag]

@giladgd alright I see there is something in the code called an abort signal, what you posted is about sessions which I did not check because I am not using sessions, and use straight generation objects without a session.

But as you say API seems alike.

I found an AbortSignal type in the types file, there is no a stopOnAbortSignal boolean, but I might assume it works the same way in this level without the boolean flag. Even the signal type has no comments or internal docs.

I will check how it behaves.

I still think it should be added somehow, I've hours lost in the documentation (mostly in API) trying to figure it out, I think that it is because most people are using a session and they'd have stumbled upon it where I passed out on session docs thinking it was irrelevant since I was not using a session.

https://node-llama-cpp.withcat.ai/guide/text-completion

Here, I'd reckon, as this is the starting point for those seeking to use llama without sessions, so those like me wouldn't get lost, wouldn't mind if it was added here?... what do you think?...

[onzag]

Alright it seems to work with that, thank you for the pointers, it also throws an error on the process, about the operation being Aborted, so you need to track the aborting process and catch the error in the error clause, to prevent an error from cascading to the client.

In my usecase, one of them for example is I have a max_paragraphs option, so as tokens are generated every time I see them "\n\n" I count a paragraph, and once I see the last, I stop the generation; but of course that isn't an error by the client's perspective; I think max_paragraphs is a great usecase that showcases using abort signals for text flow control and maybe I can add that in that part in the docs?... what do you think?...