LLM confidence threshold #16
Description
Is your feature request related to a problem? Please describe.
Some content generated by LLMs for AI code review and commit text analysis seems to lack confidence. Enforcing a "confident" tone and high confidence output are separate but should both be achieved when providing code review and commit message feedback.
There are times when generating code review on a patch with an obvious issue the model will correctly identify the issue, but when removing the issue, the model will proceed to suggesting changing other parts of the code that weren't concerning when a real issue was present. It feels like LLMs hesitate to say nothing at all, when asked to only provide critical comments or nothing at all. The model's confidence it probably low when suggesting changes when no issues are present.
Describe the solution you'd like
When receiving model responses, read their confidence and include it in the output for the review.
Additional context
The level of confidence for the model's output can be relevant for other features such as #6 where the confidence value for the generated code review can be used as a guiding metric for the intermediate assistant agent.
It could also be useful for use cases like #5 where the audience goes beyond just the developer, and only high confidence reviews are relevant.