too slow with semantic integration than directly interact with LLamaSharp

[bc4250]

Model: ggml-model-f32-q4_0.bin

with CPU backend

even just say hello, current package response with in plenty of minutes.
but if use LLamaSharp it will response immediately

just record this situation for your reference, i will dig into it if have time, maybe it because of memory limit of webapi, and console application doesn‘t have this constraint.

sample code in LLamaSharp demo

foreach (var text in session.Chat(prompt, new InferenceParams() { Temperature = 0.6f, AntiPrompts = new List<string> { "User:" } }))
    {
        Console.Write(text);
    }

the screen shot of slow reponse when using postman interact with webapi sample code。

[xbotter]

To deploy large language models in practical production, it is indeed important to pay attention to hardware resource consumption. I will continue to focus on this aspect in the future.
Recently, LLamaSharp has released a new version and I will take a look at what improvements have been made.