Implements real-time hints, a feature that allows users to provide gu… #15469
Conversation
Summary of ChangesHello @waywardgeek, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements a "real-time hints" feature, allowing users to provide immediate feedback or guidance to the AI model even when it's actively processing or executing tools. This is achieved by queuing user messages during busy periods and then injecting them into the conversation history as user messages before the model generates its next response, effectively enabling dynamic steering of the AI's behavior without interrupting its workflow. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a real-time hints feature, allowing users to provide guidance to the model mid-execution. However, a high-severity prompt injection vulnerability has been identified where user input is directly inserted into the language model's context mid-turn without sanitization, potentially allowing malicious actors to manipulate the model's behavior. Additionally, there is a high-severity issue related to error handling that could lead to application instability.
| // HINTS: Drain any queued messages as hints before sending tool responses | ||
| // Messages typed while the system is busy become hints that influence the current turn | ||
| if (popAllMessages) { | ||
| const queuedMessages = popAllMessages(); | ||
| if (queuedMessages) { | ||
| debugLogger.debug( | ||
| '[HINTS] 💧 Draining queued messages as hints:', | ||
| queuedMessages, | ||
| ); | ||
| // Add queued messages as user messages to history before tool responses | ||
| // This injects them mid-turn, allowing them to influence the model's next response | ||
| addItem( | ||
| { | ||
| type: MessageType.USER, | ||
| text: queuedMessages, | ||
| }, | ||
| Date.now(), | ||
| ); | ||
| await geminiClient.addHistory({ | ||
| role: 'user', | ||
| parts: [{ text: queuedMessages }], | ||
| }); | ||
| } | ||
| } |
There was a problem hiding this comment.
A prompt injection vulnerability exists in the handling of queued messages. User-supplied input from popAllMessages() is directly added to the chat history using geminiClient.addHistory() without sanitization. This allows a malicious user to inject arbitrary prompts, potentially manipulating the model's behavior. Additionally, the await geminiClient.addHistory() call is not wrapped in a try...catch block, which could lead to unhandled promise rejections and application instability if addHistory fails.
// HINTS: Drain any queued messages as hints before sending tool responses
// Messages typed while the system is busy become hints that influence the current turn
if (popAllMessages) {
const queuedMessages = popAllMessages();
if (queuedMessages) {
debugLogger.debug(
'[HINTS] 💧 Draining queued messages as hints:',
queuedMessages,
);
// Add queued messages as user messages to history before tool responses
// This injects them mid-turn, allowing them to influence the model's next response
const hintText = `A user has provided the following hint while you were busy: \n\n${queuedMessages}\n\nTake this into account for your next step.`;
addItem(
{
type: MessageType.USER,
text: hintText,
},
Date.now(),
);
await geminiClient.addHistory({
role: 'user',
parts: [{ text: hintText }],
});
}
}
Allows users to provide guidance to the model mid-execution without interrupting the tool chain. Messages typed while the model is busy are queued and automatically injected as user messages into the history immediately before tool responses are submitted. This enables dynamic steering: users can provide clarifications or corrections in real-time, which the model incorporates into its next response while continuing its work. Key changes: - AppContainer.tsx: Routes user input to a message queue when busy and provides a stable ref for draining. - useGeminiStream.ts: Drains queued messages and injects them as user history items before submitting tool results. - useMessageQueue.ts: Manages the lifecycle of queued messages.
waywardgeek
force-pushed
the
real-time-hints
branch
from
f2794b5 to
466da0b
Compare
1 participant
Issue #14390
Implements real-time hints, a feature that allows users to provide guidance to the AI model mid-execution without interrupting the tool chain. When users type messages while tools are running or the model is thinking, those messages are captured and automatically injected into the conversation history as user messages before the model generates its next response.
This enables dynamic steering: users can provide clarifications, corrections, or new context in real-time, and the model incorporates this feedback into its ongoing work. The tool chain continues uninterrupted, but the model's behavior adapts based on the user's hints.
How it works:
Key changes: