Incorrect endpoint being used for Local LLMs

[Saher-Anwar]

I've setup ollama (llama3) on my pc.

.env file has the following fields:

LLM_URL=http://127.0.0.1:11434
LLM_MODEL=ollama

The error:

For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
00:17:02 - INFO - [362/365] score=0  Software Developer
00:17:02 - INFO - HTTP Request: POST http://127.0.0.1:11434/chat/completions "HTTP/1.1 404 Not Found"
00:17:02 - ERROR - LLM error scoring job 'Senior Cloud Security Engineer': Client error '404 Not Found' for url 'http://127.0.0.1:11434/chat/completions'

It seems to be defaulting to the /chat/completions endpoint for local LLMs which seems to be causing the issue. On looking at the code, it seems that the assumed response will not work either.

[jayf0x]

same issue. Quick fix could be llm.py ~L168

  chat_endpoint = "api/generate" if self.base_url.startswith("http://localhost:11434") else "chat/completion"
   
resp = self._client.post(
   f"{self.base_url}/{chat_endpoint}",
    json=payload,
     headers=headers,
)

or slightly cleaner:

# L77
_OLLAMA_NATIVE_BASE = "http://localhost:11434"

# in __init__()
self._is_ollama: bool = base_url.startswith(_OLLAMA_NATIVE_BASE)
...
self.chat_endpoint = "api/generate" if self._is_ollama else "chat/completion"
self.chat_url = f"{self.base_url}/{self.chat_endpoint}"

# ~L168
resp = self._client.post(
    self.chat_url,
    json=payload,
    headers=headers,
)

note: untested.

@Saher-Anwar ps. LLM_MODEL should be the actual name of the model, like qwen3:9b.

@claude Could you assist and fact check this? Optionally also add a small sys prompt in the ask method to aid accuracy, especially for local models.